section-2-data-science-and-ml-tools/introduction-to-scikit-learn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/mrdbourke/zero-to-mastery-ml/blob/master/section-2-data-science-and-ml-tools/introduction-to-scikit-learn.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "[View source code](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/section-2-data-science-and-ml-tools/introduction-to-scikit-learn.ipynb) | [Read notebook in online book format](https://dev.mrdbourke.com/zero-to-mastery-ml/introduction-to-scikit-learn/)\n",
    "\n",
    "\n",
    "# A Quick Machine Learning Modelling Tutorial with Python and Scikit-Learn\n",
    "\n",
    "This notebook goes through a range of common and useful featues of the Scikit-Learn library.\n",
    "\n",
    "There's a bunch here but I'm calling it quick because of how vast the Scikit-Learn library is. \n",
    "\n",
    "Covering everything requires a [full-blown documentation](https://scikit-learn.org/stable/user_guide.html), of which, if you ever get stuck, I'd highly recommend checking out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last updated: 2024-09-06 13:30:34.743560\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "print(f\"Last updated: {datetime.datetime.now()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Scikit-Learn (sklearn)?\n",
    "\n",
    "[Scikit-Learn](https://scikit-learn.org/stable/index.html), also referred to as `sklearn`, is an open-source Python machine learning library.\n",
    "\n",
    "It's built on top on NumPy (Python library for numerical computing) and Matplotlib (Python library for data visualization).\n",
    "\n",
    "<img src=\"../images/sklearn-6-step-ml-framework-tools-scikit-learn-highlight.png\" alt=\"a 6 step machine learning framework along will tools you can use for each step\" width=\"700\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Scikit-Learn?\n",
    "\n",
    "Although the fields of data science and machine learning are vast, the main goal is finding patterns within data and then using those patterns to make predictions.\n",
    "\n",
    "And there are certain categories which a majority of problems fall into.\n",
    "\n",
    "If you're trying to create a machine learning model to predict whether an email is spam and or not spam, you're working on a [classification problem](https://en.wikipedia.org/wiki/Statistical_classification#Binary_and_multiclass_classification) (whether something is one thing or another).\n",
    "\n",
    "If you're trying to create a machine learning model to predict the price of houses given their characteristics, you're working on a [regression problem](https://en.wikipedia.org/wiki/Regression_analysis) (predicting a number).\n",
    "\n",
    "If you're trying to get a machine learning algorithm to group together similar samples (that you don't necessarily know which should go together), you're working on a [clustering problem](https://developers.google.com/machine-learning/clustering/overview).\n",
    "\n",
    "Once you know what kind of problem you're working on, there are also similar steps you'll take for each. \n",
    "\n",
    "Steps like splitting the data into different sets, one for your machine learning algorithms to learn on (the training set) and another to test them on (the testing set).\n",
    "\n",
    "Choosing a machine learning model and then evaluating whether or not your model has learned anything.\n",
    "\n",
    "Scikit-Learn offers Python implementations for doing all of these kinds of tasks (from preparing data to modelling data). Saving you from having to build them from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What does this notebook cover?\n",
    "\n",
    "The Scikit-Learn library is very capable. However, learning everything off by heart isn't necessary. Instead, this notebook focuses some of the main use cases of the library.\n",
    "\n",
    "More specifically, we'll cover:\n",
    "\n",
    "<img src=\"../images/sklearn-workflow-title.png\" alt=\"a 6 step scikit-learn workflow\"/>\n",
    "\n",
    "0. An end-to-end Scikit-Learn worfklow\n",
    "1. Getting the data ready\n",
    "2. Choosing the right maching learning estimator/aglorithm/model for your problem\n",
    "3. Fitting your chosen machine learning model to data and using it to make a prediction\n",
    "4. Evaluting a machine learning model\n",
    "5. Improving predictions through experimentation (hyperparameter tuning)\n",
    "6. Saving and loading a pretrained model\n",
    "7. Putting it all together in a pipeline\n",
    "\n",
    "> **Note:** All of the steps in this notebook are focused on [**supervised learning**](https://en.wikipedia.org/wiki/Supervised_learning) (having data and labels). The other side of supervised learning is [**unsupervised learning**](https://en.wikipedia.org/wiki/Unsupervised_learning) (having data but no labels). \n",
    "\n",
    "After going through it, you'll have the base knolwedge of Scikit-Learn you need to keep moving forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where can I get help?\n",
    "\n",
    "If you get stuck or think of something you'd like to do which this notebook doesn't cover, don't fear!\n",
    "\n",
    "The recommended steps you take are:\n",
    "1. **Try it** - Since Scikit-Learn has been designed with usability in mind, your first step should be to use what you know and try figure out the answer to your own question (getting it wrong is part of the process). If in doubt, run your code.\n",
    "2. **Press SHIFT+TAB** - See you can the docstring of a function (information on what the function does) by pressing **SHIFT + TAB** inside it. Doing this is a good habit to develop. It'll improve your research skills and give you a better understanding of the library. \n",
    "3. **Search for it** - If trying it on your own doesn't work, since someone else has probably tried to do something similar, try searching for your problem. You'll likely end up in 1 of 2 places:\n",
    "    * [Scikit-Learn documentation/user guide](https://scikit-learn.org/stable/user_guide.html) - the most extensive resource you'll find for Scikit-Learn information.\n",
    "    * [Stack Overflow](https://stackoverflow.com/) - this is the developers Q&A hub, it's full of questions and answers of different problems across a wide range of software development topics and chances are, there's one related to your problem.\n",
    "    * [ChatGPT](https://chat.openai.com/) - ChatGPT is very good at explaining code, however, it can make mistakes. Best to verify the code it writes first before using it. Try asking \"Can you explain the following code for me? {your code here}\" and then continue with follow up questions from there. Avoid blindly trusting code you didn't write for yourself.\n",
    "    \n",
    "An example of searching for a Scikit-Learn solution might be:\n",
    "\n",
    "> \"how to tune the hyperparameters of a sklearn model\"\n",
    "\n",
    "Searching this on Google leads to the Scikit-Learn documentation for the `GridSearchCV` function: http://scikit-learn.org/stable/modules/grid_search.html\n",
    "\n",
    "The next steps here are to read through the documentation, check the examples and see if they line up to the problem you're trying to solve. If they do, **rewrite the code** to suit your needs, run it, and see what the outcomes are.\n",
    "\n",
    "4. **Ask for help** - If you've been through the above 3 steps and you're still stuck, you might want to ask your question on [Stack Overflow](https://www.stackoverflow.com) or in the ZTM Machine Learning and AI Discord channel. Be as specific as possible and provide details on what you've tried.\n",
    "\n",
    "Remember, you don't have to learn all of the functions off by heart to begin with. \n",
    "\n",
    "What's most important is continually asking yourself, \"what am I trying to do with the data?\".\n",
    "\n",
    "Start by answering that question and then practicing finding the code which does it.\n",
    "\n",
    "Let's get started.\n",
    "\n",
    "First we'll import the libraries we've been using previously.\n",
    "\n",
    "We'll also check the version of `sklearn` we've got."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Scikit-Learn version: 1.5.1 (materials in this notebook require this version or newer).\n"
     ]
    }
   ],
   "source": [
    "# Standard imports\n",
    "# %matplotlib inline # No longer required in newer versions of Jupyter (2022+)\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import sklearn\n",
    "print(f\"Using Scikit-Learn version: {sklearn.__version__} (materials in this notebook require this version or newer).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. An end-to-end Scikit-Learn workflow\n",
    "\n",
    "Before we get in-depth, let's quickly check out what an end-to-end Scikit-Learn workflow might look like.\n",
    "\n",
    "Once we've seen an end-to-end workflow, we'll dive into each step a little deeper.\n",
    "\n",
    "Specifically, we'll get hands-on with the following steps:\n",
    "1. Getting data ready (split into features and labels, prepare train and test steps)\n",
    "2. Choosing a model for our problem\n",
    "3. Fit the model to the data and use it to make a prediction\n",
    "4. Evaluate the model\n",
    "5. Experiment to improve\n",
    "6. Save a model for someone else to use\n",
    "\n",
    "> **Note:** The following section is a bit information heavy but it is an end-to-end workflow. We'll go through it quite swiftly but we'll break it down more throughout the rest of the notebook. And since Scikit-Learn is such a vast library, capable of tackling many problems, the workflow we're using is only one example of how you can use it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Classifier Workflow for Classifying Heart Disease\n",
    "\n",
    "#### 1. Get the data ready\n",
    "\n",
    "As an example dataset, we'll import `heart-disease.csv`.\n",
    "\n",
    "This file contains anonymised patient medical records and whether or not they have heart disease or not (this is a classification problem since we're trying to predict whether something is one thing or another)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   0     1       1  \n",
       "1   0     2       1  \n",
       "2   0     2       1  \n",
       "3   0     2       1  \n",
       "4   0     2       1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# heart_disease = pd.read_csv(\"../data/heart-disease.csv\") # load data from local directory\n",
    "heart_disease = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\") # load data directly from URL (requires raw form on GitHub, source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/heart-disease.csv)\n",
    "heart_disease.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, each row is a different patient and all columns except `target` are different patient characteristics. \n",
    "\n",
    "The `target` column indicates whether the patient has heart disease (`target=1`) or not (`target=0`), this is our \"label\" columnm, the variable we're going to try and predict.\n",
    "\n",
    "The rest of the columns (often called features) are what we'll be using to predict the `target` value.\n",
    "\n",
    "> **Note:** It's a common custom to save features to a varialbe `X` and labels to a variable `y`. In practice, we'd like to use the `X` (features) to build a predictive algorithm to predict the `y` (labels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  \n",
       "0   0     1  \n",
       "1   0     2  \n",
       "2   0     2  \n",
       "3   0     2  \n",
       "4   0     2  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create X (all the feature columns)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "\n",
    "# Create y (the target column)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Check the head of the features DataFrame\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0    1\n",
       " 1    1\n",
       " 2    1\n",
       " 3    1\n",
       " 4    1\n",
       " Name: target, dtype: int64,\n",
       " target\n",
       " 1    165\n",
       " 0    138\n",
       " Name: count, dtype: int64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the head and the value counts of the labels \n",
    "y.head(), y.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most important practices in machine learning is to split datasets into training and test sets.\n",
    "\n",
    "As in, a model will **train on the training set** to learn patterns and then those patterns can be **evaluated on the test set**.\n",
    "\n",
    "Crucially, a model should **never** see testing data during training.\n",
    "\n",
    "This is equivalent to a student studying course materials during the semester (training set) and then testing their abilities on the following exam (testing set).\n",
    "\n",
    "Scikit-learn provides the [`sklearn.model_selection.train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) method to split datasets in training and test sets.\n",
    "\n",
    "> **Note:** A common practice to use an 80/20 or 70/30 or 75/25 split for training/testing data. There is also a third set, known as a validation set (e.g. 70/15/15 for training/validation/test) for hyperparamter tuning on but for now we'll focus on training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((227, 13), (76, 13), (227,), (76,))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split the data into training and test sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y,\n",
    "                                                    test_size=0.25) # by default train_test_split uses 25% of the data for the test set\n",
    "\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Choose the model and hyperparameters\n",
    "\n",
    "Choosing a model often depends on the type of problem you're working on.\n",
    "\n",
    "For example, there are different models that Scikit-Learn recommends whether you're working on a classification or regression problem.\n",
    "\n",
    "You can see a map breaking down the [different kinds of model options and recommendations in the Scikit-Learn documentation](https://scikit-learn.org/stable/machine_learning_map.html).\n",
    "\n",
    "Scikit-Learn refers to models as \"estimators\", however, they are often also referred to as `model` or `clf` (short for classifier).\n",
    "\n",
    "A model's hyperparameters are settings you can change to adjust it for your problem, much like knobs on an oven you can tune to cook your favourite dish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we're working on a classification problem, we'll start with a RandomForestClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the current hyperparameters of a model with the [`get_params()`](https://scikit-learn.org/stable/developers/develop.html#get-params-and-set-params) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'class_weight': None,\n",
       " 'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'monotonic_cst': None,\n",
       " 'n_estimators': 100,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the current hyperparameters\n",
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll leave this as is for now, as Scikit-Learn models generally have good default settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Fit the model to the data and use it to make a prediction\n",
    "\n",
    "Fitting a model a dataset involves passing it the data and asking it to figure out the patterns.\n",
    "\n",
    "If there are labels (supervised learning), the model tries to work out the relationship between the data and the labels. \n",
    "\n",
    "If there are no labels (unsupervised learning), the model tries to find patterns and group similar samples together.\n",
    "\n",
    "Most Scikit-Learn models have the [`fit(X, y)`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.fit) method built-in, where the `X` parameter is the features and the `y` parameter is the labels.\n",
    "\n",
    "In our case, we start by fitting a model on the training split (`X_train`, `y_train`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestClassifier()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;RandomForestClassifier<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestClassifier.html\">?<span>Documentation for RandomForestClassifier</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestClassifier()</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestClassifier()"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the model to make a prediction\n",
    "\n",
    "The whole point of training a machine learning model is to use it to make some kind of prediction in the future.\n",
    "\n",
    "Once your model instance is trained, you can use the [`predict()`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict) method to predict a target value given a set of features. \n",
    "\n",
    "In other words, use the model, along with some new, unseen and unlabelled data to predict the label.\n",
    "\n",
    "> **Note:** Data you predict on should be in the same shape and format as data you trained on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/daniel/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py:493: UserWarning: X does not have valid feature names, but RandomForestClassifier was fitted with feature names\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "Expected 2D array, got 1D array instead:\narray=[0. 2. 3. 4.].\nReshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[10], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# This doesn't work... incorrect shapes\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m y_label \u001b[38;5;241m=\u001b[39m \u001b[43mclf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpredict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43marray\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m3\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m4\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py:904\u001b[0m, in \u001b[0;36mForestClassifier.predict\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    883\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpredict\u001b[39m(\u001b[38;5;28mself\u001b[39m, X):\n\u001b[1;32m    884\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    885\u001b[0m \u001b[38;5;124;03m    Predict class for X.\u001b[39;00m\n\u001b[1;32m    886\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    902\u001b[0m \u001b[38;5;124;03m        The predicted classes.\u001b[39;00m\n\u001b[1;32m    903\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 904\u001b[0m     proba \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpredict_proba\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    906\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mn_outputs_ \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m    907\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclasses_\u001b[38;5;241m.\u001b[39mtake(np\u001b[38;5;241m.\u001b[39margmax(proba, axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m), axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py:946\u001b[0m, in \u001b[0;36mForestClassifier.predict_proba\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    944\u001b[0m check_is_fitted(\u001b[38;5;28mself\u001b[39m)\n\u001b[1;32m    945\u001b[0m \u001b[38;5;66;03m# Check data\u001b[39;00m\n\u001b[0;32m--> 946\u001b[0m X \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_validate_X_predict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    948\u001b[0m \u001b[38;5;66;03m# Assign chunk of trees to jobs\u001b[39;00m\n\u001b[1;32m    949\u001b[0m n_jobs, _, _ \u001b[38;5;241m=\u001b[39m _partition_estimators(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mn_estimators, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mn_jobs)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py:641\u001b[0m, in \u001b[0;36mBaseForest._validate_X_predict\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    638\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    639\u001b[0m     force_all_finite \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 641\u001b[0m X \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_validate_data\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    642\u001b[0m \u001b[43m    \u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    643\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mDTYPE\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    644\u001b[0m \u001b[43m    \u001b[49m\u001b[43maccept_sparse\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcsr\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    645\u001b[0m \u001b[43m    \u001b[49m\u001b[43mreset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    646\u001b[0m \u001b[43m    \u001b[49m\u001b[43mforce_all_finite\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_all_finite\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    647\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    648\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m issparse(X) \u001b[38;5;129;01mand\u001b[39;00m (X\u001b[38;5;241m.\u001b[39mindices\u001b[38;5;241m.\u001b[39mdtype \u001b[38;5;241m!=\u001b[39m np\u001b[38;5;241m.\u001b[39mintc \u001b[38;5;129;01mor\u001b[39;00m X\u001b[38;5;241m.\u001b[39mindptr\u001b[38;5;241m.\u001b[39mdtype \u001b[38;5;241m!=\u001b[39m np\u001b[38;5;241m.\u001b[39mintc):\n\u001b[1;32m    649\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNo support for np.int64 index based sparse matrices\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py:633\u001b[0m, in \u001b[0;36mBaseEstimator._validate_data\u001b[0;34m(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)\u001b[0m\n\u001b[1;32m    631\u001b[0m         out \u001b[38;5;241m=\u001b[39m X, y\n\u001b[1;32m    632\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m no_val_X \u001b[38;5;129;01mand\u001b[39;00m no_val_y:\n\u001b[0;32m--> 633\u001b[0m     out \u001b[38;5;241m=\u001b[39m \u001b[43mcheck_array\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minput_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mX\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mcheck_params\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    634\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m no_val_X \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m no_val_y:\n\u001b[1;32m    635\u001b[0m     out \u001b[38;5;241m=\u001b[39m _check_y(y, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mcheck_params)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:1050\u001b[0m, in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001b[0m\n\u001b[1;32m   1043\u001b[0m         \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1044\u001b[0m             msg \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m   1045\u001b[0m                 \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mExpected 2D array, got 1D array instead:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124marray=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00marray\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1046\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mReshape your data either using array.reshape(-1, 1) if \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1047\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myour data has a single feature or array.reshape(1, -1) \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1048\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mif it contains a single sample.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1049\u001b[0m             )\n\u001b[0;32m-> 1050\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(msg)\n\u001b[1;32m   1052\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m dtype_numeric \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(array\u001b[38;5;241m.\u001b[39mdtype, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mkind\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;129;01mand\u001b[39;00m array\u001b[38;5;241m.\u001b[39mdtype\u001b[38;5;241m.\u001b[39mkind \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mUSV\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[1;32m   1053\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1054\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdtype=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mnumeric\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m is not compatible with arrays of bytes/strings.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1055\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mConvert your data to numeric values explicitly instead.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1056\u001b[0m     )\n",
      "\u001b[0;31mValueError\u001b[0m: Expected 2D array, got 1D array instead:\narray=[0. 2. 3. 4.].\nReshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample."
     ]
    }
   ],
   "source": [
    "# This doesn't work... incorrect shapes\n",
    "y_label = clf.predict(np.array([0, 2, 3, 4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! \n",
    "\n",
    "We get a `ValueError` (mismatched shapes):\n",
    "\n",
    "```\n",
    "ValueError: Expected 2D array, got 1D array instead:\n",
    "array=[0. 2. 3. 4.].\n",
    "Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.\n",
    "```\n",
    "\n",
    "This happens because we're **trying to make predictions on data that is in a different format to the data our model was trained on**.\n",
    "\n",
    "Since our model was trained on data from `X_train`, predictions should be made on data in the same format and shape as `X_train`.\n",
    "\n",
    "Our goal in many machine learning problems is to use patterns learned from the training data to make predictions on the test data (or future unseen data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>227</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>154</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>40</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>140</td>\n",
       "      <td>199</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>1</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>69</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>140</td>\n",
       "      <td>239</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>151</td>\n",
       "      <td>0</td>\n",
       "      <td>1.8</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>244</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>154</td>\n",
       "      <td>1</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "165   67    1   0       160   286    0        0      108      1      1.5   \n",
       "71    51    1   2        94   227    0        1      154      1      0.0   \n",
       "24    40    1   3       140   199    0        1      178      1      1.4   \n",
       "19    69    0   3       140   239    0        1      151      0      1.8   \n",
       "258   62    0   0       150   244    0        1      154      1      1.4   \n",
       "\n",
       "     slope  ca  thal  \n",
       "165      1   3     2  \n",
       "71       2   1     3  \n",
       "24       2   0     3  \n",
       "19       2   2     2  \n",
       "258      1   0     2  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In order to predict a label, data has to be in the same shape as X_train\n",
    "X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the model to make a prediction on the test data (further evaluation)\n",
    "y_preds = clf.predict(X=X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Evaluate the model\n",
    "\n",
    "Now we've made some predictions, we can start to use some more Scikit-Learn methods to figure out how good our model is. \n",
    "\n",
    "Each model or estimator has a built-in [`score()`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.score) method. \n",
    "\n",
    "This method compares how well the model was able to learn the patterns between the features and labels.\n",
    "\n",
    "The `score()` method for each model uses a standard evaluation metric to measure your model's results.\n",
    "\n",
    "In the case of a classifier (our model), one of the most common evaluation metrics is [accuracy](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score) (the fraction of correct predictions out of total predictions).\n",
    "\n",
    "Let's check out our model's accuracy on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model's accuracy on the training dataset is: 100.0%\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model on the training set\n",
    "train_acc = clf.score(X=X_train, y=y_train)\n",
    "print(f\"The model's accuracy on the training dataset is: {train_acc*100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! Looks like our model does pretty well on the training datset.\n",
    "\n",
    "This is because it has a chance to see both data *and* labels.\n",
    "\n",
    "How about the test dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model's accuracy on the testing dataset is: 88.16%\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model on the test set\n",
    "test_acc = clf.score(X=X_test, y=y_test)\n",
    "print(f\"The model's accuracy on the testing dataset is: {test_acc*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, looks like our model's accuracy is a bit less on the test dataset than the training dataset.\n",
    "\n",
    "This is quite often the case, because remember, a model has never seen the testing examples before.\n",
    "\n",
    "There are also a number of other evaluation methods we can use for our classification models.\n",
    "\n",
    "All of the following classification metrics come from the [`sklearn.metrics`](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics) module:\n",
    "* [`classification_report(y_true, y_true)`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report) - Builds a text report showing various classification metrics such as [precision, recall](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html) and F1-score.\n",
    "* [`confusion_matrix(y_true, y_pred)`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix) - Create a [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) to compare predictions to truth labels.\n",
    "* [`accuracy_score(y_true, y_pred)`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) - Find the accuracy score (the default metric) for a classifier.\n",
    "\n",
    "All metrics have the following in common: they compare a model's predictions (`y_pred`) to truth labels (`y_true`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.84      0.85        31\n",
      "           1       0.89      0.91      0.90        45\n",
      "\n",
      "    accuracy                           0.88        76\n",
      "   macro avg       0.88      0.87      0.88        76\n",
      "weighted avg       0.88      0.88      0.88        76\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n",
    "\n",
    "# Create a classification report\n",
    "print(classification_report(y_test, y_preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[26,  5],\n",
       "       [ 4, 41]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a confusion matrix\n",
    "conf_mat = confusion_matrix(y_test, y_preds)\n",
    "conf_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.881578947368421"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the accuracy score (same as the score() method for classifiers) \n",
    "accuracy_score(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Experiment to improve\n",
    "\n",
    "The first model you build is often referred to as a baseline (a baseline is often even simpler than the model we've used, a baseline could be \"let's just by default predict the most common value and then try to improve\").\n",
    "\n",
    "Once you've got a baseline model, like we have here, it's important to remember, this is often not the final model you'll use.\n",
    "\n",
    "The next step in the workflow is to try and improve upon your baseline model.\n",
    "\n",
    "How?\n",
    "\n",
    "With one of the most important mottos in machine learning...\n",
    "\n",
    "*Experiment, experiment, experiment!*\n",
    "\n",
    "Experiments can come in many different forms.\n",
    "\n",
    "But let's break it into two.\n",
    "\n",
    "1. From a model perspective.\n",
    "2. From a data perspective.\n",
    "\n",
    "From a model perspective may involve things such as using a more complex model or tuning your models hyperparameters.\n",
    "\n",
    "From a data perspective may involve collecting more data or better quality data so your existing model has more of a chance to learn the patterns within.\n",
    "\n",
    "If you're already working on an existing dataset, it's often easier try a series of model perspective experiments first and then turn to data perspective experiments if you aren't getting the results you're looking for.\n",
    "\n",
    "One thing you should be aware of is if you're tuning a models hyperparameters in a series of experiments, your reuslts should always be cross-validated (we'll see this later on!). \n",
    "\n",
    "[Cross-validation](http://scikit-learn.org/stable/modules/cross_validation.html) is a way of making sure the results you're getting are consistent across your training and test datasets (because it uses multiple versions of training and test sets) rather than just luck because of the order the original training and test sets were created. \n",
    "\n",
    "* Try different hyperparameters.\n",
    "* All different parameters should be cross-validated.\n",
    "    * **Note:** Beware of cross-validation for time series problems (as for time series, you don't want to mix samples from the future with samples from the past).\n",
    "    \n",
    "Different models you use will have different hyperparameters you can tune. \n",
    "\n",
    "For the case of our model, the [`RandomForestClassifier()`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), we'll start trying different values for `n_estimators` (a measure for the number of trees in the random forest).\n",
    "\n",
    "By default, `n_estimators=100`, so how about we try values from `100` to `200` and see what happens (generally more is better)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying model with 100 estimators...\n",
      "Model accuracy on test set: 88.16%\n",
      "\n",
      "Trying model with 110 estimators...\n",
      "Model accuracy on test set: 90.79%\n",
      "\n",
      "Trying model with 120 estimators...\n",
      "Model accuracy on test set: 90.79%\n",
      "\n",
      "Trying model with 130 estimators...\n",
      "Model accuracy on test set: 89.47%\n",
      "\n",
      "Trying model with 140 estimators...\n",
      "Model accuracy on test set: 88.16%\n",
      "\n",
      "Trying model with 150 estimators...\n",
      "Model accuracy on test set: 94.74%\n",
      "\n",
      "Trying model with 160 estimators...\n",
      "Model accuracy on test set: 92.11%\n",
      "\n",
      "Trying model with 170 estimators...\n",
      "Model accuracy on test set: 92.11%\n",
      "\n",
      "Trying model with 180 estimators...\n",
      "Model accuracy on test set: 92.11%\n",
      "\n",
      "Trying model with 190 estimators...\n",
      "Model accuracy on test set: 89.47%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Try different numbers of estimators (trees)... (no cross-validation)\n",
    "np.random.seed(42)\n",
    "for i in range(100, 200, 10):\n",
    "    print(f\"Trying model with {i} estimators...\")\n",
    "    model = RandomForestClassifier(n_estimators=i).fit(X_train, y_train)\n",
    "    print(f\"Model accuracy on test set: {model.score(X_test, y_test) * 100:.2f}%\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metrics above were measured on a single train and test split.\n",
    "\n",
    "Let's use [`sklearn.model_selection.cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) to measure the results across 5 different train and test sets.\n",
    "\n",
    "We can achieve this by setting `cross_val_score(X, y, cv=5)`.\n",
    "\n",
    "Where `X` is the *full* feature set and `y` is the *full* label set and `cv` is the number of train and test splits `cross_val_score` will automatically create from the data (in our case, `5` different splits, this is known as 5-fold cross-validation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying model with 100 estimators...\n",
      "Model accuracy on single test set split: 88.16%\n",
      "5-fold cross-validation score: 82.15%\n",
      "\n",
      "Trying model with 110 estimators...\n",
      "Model accuracy on single test set split: 94.74%\n",
      "5-fold cross-validation score: 81.17%\n",
      "\n",
      "Trying model with 120 estimators...\n",
      "Model accuracy on single test set split: 88.16%\n",
      "5-fold cross-validation score: 83.49%\n",
      "\n",
      "Trying model with 130 estimators...\n",
      "Model accuracy on single test set split: 89.47%\n",
      "5-fold cross-validation score: 83.14%\n",
      "\n",
      "Trying model with 140 estimators...\n",
      "Model accuracy on single test set split: 88.16%\n",
      "5-fold cross-validation score: 82.48%\n",
      "\n",
      "Trying model with 150 estimators...\n",
      "Model accuracy on single test set split: 90.79%\n",
      "5-fold cross-validation score: 80.17%\n",
      "\n",
      "Trying model with 160 estimators...\n",
      "Model accuracy on single test set split: 89.47%\n",
      "5-fold cross-validation score: 80.83%\n",
      "\n",
      "Trying model with 170 estimators...\n",
      "Model accuracy on single test set split: 86.84%\n",
      "5-fold cross-validation score: 82.16%\n",
      "\n",
      "Trying model with 180 estimators...\n",
      "Model accuracy on single test set split: 92.11%\n",
      "5-fold cross-validation score: 81.50%\n",
      "\n",
      "Trying model with 190 estimators...\n",
      "Model accuracy on single test set split: 88.16%\n",
      "5-fold cross-validation score: 81.83%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# With cross-validation\n",
    "np.random.seed(42)\n",
    "for i in range(100, 200, 10):\n",
    "    print(f\"Trying model with {i} estimators...\")\n",
    "    model = RandomForestClassifier(n_estimators=i).fit(X_train, y_train)\n",
    "\n",
    "    # Measure the model score on a single train/test split\n",
    "    model_score = model.score(X_test, y_test)\n",
    "    print(f\"Model accuracy on single test set split: {model_score * 100:.2f}%\")\n",
    "    \n",
    "    # Measure the mean cross-validation score across 5 different train and test splits\n",
    "    cross_val_mean = np.mean(cross_val_score(model, X, y, cv=5))\n",
    "    print(f\"5-fold cross-validation score: {cross_val_mean * 100:.2f}%\")\n",
    "    \n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which model had the best cross-validation score?\n",
    "\n",
    "This is usually a better indicator of a quality model than a single split accuracy score.\n",
    "\n",
    "Rather than set up and track the results of these experiments manually, we can get Scikit-Learn to do the exploration for us.\n",
    "\n",
    "Scikit-Learn's [`sklearn.model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) is a way to search over a set of different hyperparameter values and automatically track which perform the best.\n",
    "\n",
    "Let's test it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "The best parameter values are: {'n_estimators': 120}\n",
      "With a score of: 82.82%\n"
     ]
    }
   ],
   "source": [
    "# Another way to do it with GridSearchCV...\n",
    "np.random.seed(42)\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Define the parameters to search over in dictionary form \n",
    "# (these can be any of your target model's hyperparameters) \n",
    "param_grid = {'n_estimators': [i for i in range(100, 200, 10)]}\n",
    "\n",
    "# Setup the grid search\n",
    "grid = GridSearchCV(estimator=RandomForestClassifier(),\n",
    "                    param_grid=param_grid,\n",
    "                    cv=5,\n",
    "                    verbose=1) \n",
    "\n",
    "# Fit the grid search to the data\n",
    "grid.fit(X, y)\n",
    "\n",
    "# Find the best parameters\n",
    "print(f\"The best parameter values are: {grid.best_params_}\")\n",
    "print(f\"With a score of: {grid.best_score_*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the best model/estimator with the `best_estimator_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestClassifier(n_estimators=120)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;RandomForestClassifier<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestClassifier.html\">?<span>Documentation for RandomForestClassifier</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestClassifier(n_estimators=120)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestClassifier(n_estimators=120)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the model to be the best estimator\n",
    "clf = grid.best_estimator_\n",
    "clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we've got the best cross-validated model, we can fit and score it on our original single train/test split of the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model score on single split of the data: 85.53%\n"
     ]
    }
   ],
   "source": [
    "# Fit the best model\n",
    "clf = clf.fit(X_train, y_train)\n",
    "\n",
    "# Find the best model scores on our single test split\n",
    "# (note: this may be lower than the cross-validation score since it's only on one split of the data)\n",
    "print(f\"Best model score on single split of the data: {clf.score(X_test, y_test)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Save a model for someone else to use\n",
    "\n",
    "When you've done a few experiments and you're happy with how your model is doing, you'll likely want someone else to be able to use it.\n",
    "\n",
    "This may come in the form of a teammate or colleague trying to replicate and validate your results or through a customer using your model as part of a service or application you offer.\n",
    "\n",
    "Saving a model also allows you to reuse it later without having to go through retraining it. Which is helpful, especially when your training times start to increase.\n",
    "\n",
    "You can [save a Scikit-Learn model](https://scikit-learn.org/stable/model_persistence.html) using Python's in-built [`pickle` module](https://docs.python.org/3/library/pickle.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Save an existing model to file\n",
    "pickle.dump(model, open(\"random_forest_model_1.pkl\", \"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded pickle model prediction score: 88.16%\n"
     ]
    }
   ],
   "source": [
    "# Load a saved pickle model and evaluate it\n",
    "loaded_pickle_model = pickle.load(open(\"random_forest_model_1.pkl\", \"rb\"))\n",
    "print(f\"Loaded pickle model prediction score: {loaded_pickle_model.score(X_test, y_test) * 100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For larger models, it may be more efficient to use [Joblib](https://joblib.readthedocs.io/en/stable/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['random_forest_model_1.joblib']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from joblib import dump, load\n",
    "\n",
    "# Save a model using joblib\n",
    "dump(model, \"random_forest_model_1.joblib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded joblib model prediction score: 88.16%\n"
     ]
    }
   ],
   "source": [
    "# Load a saved joblib model and evaluate it\n",
    "loaded_joblib_model = load(\"random_forest_model_1.joblib\")\n",
    "print(f\"Loaded joblib model prediction score: {loaded_joblib_model.score(X_test, y_test) * 100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah!\n",
    "\n",
    "We've covered a lot of ground fast...\n",
    "\n",
    "Let's break things down a bit more by revisting each section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Getting the data ready\n",
    "\n",
    "Data doesn't always come ready to use with a Scikit-Learn machine learning model.\n",
    "\n",
    "Three of the main steps you'll often have to take are:\n",
    "* Splitting the data into features (usually `X`) and labels (usually `y`).\n",
    "* Splitting the data into training and testing sets (and possibly a validation set).\n",
    "* Filling (also called imputing) or disregarding missing values.\n",
    "* Converting non-numerical values to numerical values (also call feature encoding).\n",
    "\n",
    "Let's see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   0     1       1  \n",
       "1   0     2       1  \n",
       "2   0     2       1  \n",
       "3   0     2       1  \n",
       "4   0     2       1  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Splitting the data into X & y\n",
    "heart_disease.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>140</td>\n",
       "      <td>241</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>123</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>110</td>\n",
       "      <td>264</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>132</td>\n",
       "      <td>0</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>68</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>144</td>\n",
       "      <td>193</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>141</td>\n",
       "      <td>0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>130</td>\n",
       "      <td>131</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>115</td>\n",
       "      <td>1</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>174</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>303 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0     63    1   3       145   233    1        0      150      0      2.3   \n",
       "1     37    1   2       130   250    0        1      187      0      3.5   \n",
       "2     41    0   1       130   204    0        0      172      0      1.4   \n",
       "3     56    1   1       120   236    0        1      178      0      0.8   \n",
       "4     57    0   0       120   354    0        1      163      1      0.6   \n",
       "..   ...  ...  ..       ...   ...  ...      ...      ...    ...      ...   \n",
       "298   57    0   0       140   241    0        1      123      1      0.2   \n",
       "299   45    1   3       110   264    0        1      132      0      1.2   \n",
       "300   68    1   0       144   193    1        1      141      0      3.4   \n",
       "301   57    1   0       130   131    0        1      115      1      1.2   \n",
       "302   57    0   1       130   236    0        0      174      0      0.0   \n",
       "\n",
       "     slope  ca  thal  \n",
       "0        0   0     1  \n",
       "1        0   0     2  \n",
       "2        2   0     2  \n",
       "3        2   0     2  \n",
       "4        2   0     2  \n",
       "..     ...  ..   ...  \n",
       "298      1   0     3  \n",
       "299      1   0     3  \n",
       "300      1   2     3  \n",
       "301      1   1     3  \n",
       "302      1   1     2  \n",
       "\n",
       "[303 rows x 13 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Splitting the data into features (X) and labels (y)\n",
    "X = heart_disease.drop('target', axis=1)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Looks like our dataset has 303 samples with 13 features (13 columns).\n",
    "\n",
    "Let's check out the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      1\n",
       "1      1\n",
       "2      1\n",
       "3      1\n",
       "4      1\n",
       "      ..\n",
       "298    0\n",
       "299    0\n",
       "300    0\n",
       "301    0\n",
       "302    0\n",
       "Name: target, Length: 303, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = heart_disease['target']\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful, 303 labels with values of `0` (no heart disease) and `1` (heart disease).\n",
    "\n",
    "Now let's split our data into training and test sets, we'll use an 80/20 split (80% of samples for training and 20% of samples for testing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((242, 13), (61, 13), (242,), (61,))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Splitting the data into training and test sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y, \n",
    "                                                    test_size=0.2) # you can change the test size\n",
    "\n",
    "# Check the shapes of different data splits\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "242.4"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 80% of data is being used for the training set (the model will learn patterns on these samples)\n",
    "X.shape[0] * 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60.6"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And 20% of the data is being used for the testing set (the model will be evaluated on these samples)\n",
    "X.shape[0] * 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Make sure it's all numerical\n",
    "\n",
    "Computers love numbers.\n",
    "\n",
    "So one thing you'll often have to make sure of is that your datasets are in numerical form.\n",
    "\n",
    "This even goes for datasets which contain non-numerical features that you may want to include in a model.\n",
    "\n",
    "For example, if we were working with a car sales dataset, how might we turn features such as `Make` and `Colour` into numbers?\n",
    "\n",
    "Let's figure it out.\n",
    "\n",
    "First, we'll import the [`car-sales-extended.csv`](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended.csv) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431</td>\n",
       "      <td>4</td>\n",
       "      <td>15323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714</td>\n",
       "      <td>5</td>\n",
       "      <td>19943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714</td>\n",
       "      <td>4</td>\n",
       "      <td>28343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365</td>\n",
       "      <td>4</td>\n",
       "      <td>13434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577</td>\n",
       "      <td>3</td>\n",
       "      <td>14043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>Black</td>\n",
       "      <td>35820</td>\n",
       "      <td>4</td>\n",
       "      <td>32042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>White</td>\n",
       "      <td>155144</td>\n",
       "      <td>3</td>\n",
       "      <td>5716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>66604</td>\n",
       "      <td>4</td>\n",
       "      <td>31570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>215883</td>\n",
       "      <td>4</td>\n",
       "      <td>4001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>Blue</td>\n",
       "      <td>248360</td>\n",
       "      <td>4</td>\n",
       "      <td>12732</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Make Colour  Odometer (KM)  Doors  Price\n",
       "0     Honda  White          35431      4  15323\n",
       "1       BMW   Blue         192714      5  19943\n",
       "2     Honda  White          84714      4  28343\n",
       "3    Toyota  White         154365      4  13434\n",
       "4    Nissan   Blue         181577      3  14043\n",
       "..      ...    ...            ...    ...    ...\n",
       "995  Toyota  Black          35820      4  32042\n",
       "996  Nissan  White         155144      3   5716\n",
       "997  Nissan   Blue          66604      4  31570\n",
       "998   Honda  White         215883      4   4001\n",
       "999  Toyota   Blue         248360      4  12732\n",
       "\n",
       "[1000 rows x 5 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import car-sales-extended.csv\n",
    "# car_sales = pd.read_csv(\"../data/car-sales-extended.csv\") # load data from local directory \n",
    "car_sales = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended.csv\") # load data directly from raw URL (source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended.csv)\n",
    "car_sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the dataset types with `.dtypes`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             object\n",
       "Colour           object\n",
       "Odometer (KM)     int64\n",
       "Doors             int64\n",
       "Price             int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car_sales.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the `Make` and `Colour` features are of `dtype=object` (they're strings) where as the rest of the columns are of `dtype=int64`.\n",
    "\n",
    "If we want to use the `Make` and `Colour` features in our model, we'll need to figure out how to turn them into numerical form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split into X & y and train/test\n",
    "X = car_sales.drop(\"Price\", axis=1)\n",
    "y = car_sales[\"Price\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try and build a model on our `car_sales` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "could not convert string to float: 'Honda'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/c4/qj4gdk190td18bqvjjh0p3p00000gn/T/ipykernel_23180/1044518071.py\u001b[0m in \u001b[0;36m?\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Try to predict with random forest on price column (doesn't work)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensemble\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mRandomForestRegressor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRandomForestRegressor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(estimator, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1469\u001b[0m                 skip_parameter_validation=(\n\u001b[1;32m   1470\u001b[0m                     \u001b[0mprefer_skip_nested_validation\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mglobal_skip_validation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1471\u001b[0m                 )\n\u001b[1;32m   1472\u001b[0m             ):\n\u001b[0;32m-> 1473\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfit_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    359\u001b[0m         \u001b[0;31m# Validate or convert input data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0missparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    361\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"sparse multilabel-indicator for y is not supported.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    362\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 363\u001b[0;31m         X, y = self._validate_data(\n\u001b[0m\u001b[1;32m    364\u001b[0m             \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    365\u001b[0m             \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    366\u001b[0m             \u001b[0mmulti_output\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)\u001b[0m\n\u001b[1;32m    646\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;34m\"estimator\"\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mcheck_y_params\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    647\u001b[0m                     \u001b[0mcheck_y_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mdefault_check_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mcheck_y_params\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    648\u001b[0m                 \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"y\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mcheck_y_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    649\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 650\u001b[0;31m                 \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_X_y\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mcheck_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    651\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    652\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    653\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mno_val_X\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mcheck_params\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ensure_2d\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1297\u001b[0m         raise ValueError(\n\u001b[1;32m   1298\u001b[0m             \u001b[0;34mf\"{estimator_name} requires y to be passed, but the target y is None\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1299\u001b[0m         )\n\u001b[1;32m   1300\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1301\u001b[0;31m     X = check_array(\n\u001b[0m\u001b[1;32m   1302\u001b[0m         \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1303\u001b[0m         \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0maccept_sparse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1304\u001b[0m         \u001b[0maccept_large_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0maccept_large_sparse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001b[0m\n\u001b[1;32m   1009\u001b[0m                         )\n\u001b[1;32m   1010\u001b[0m                     \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mxp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1011\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1012\u001b[0m                     \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_asarray_with_order\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mxp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1013\u001b[0;31m             \u001b[0;32mexcept\u001b[0m \u001b[0mComplexWarning\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mcomplex_warning\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1014\u001b[0m                 raise ValueError(\n\u001b[1;32m   1015\u001b[0m                     \u001b[0;34m\"Complex data not supported\\n{}\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1016\u001b[0m                 ) from complex_warning\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/_array_api.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(array, dtype, order, copy, xp, device)\u001b[0m\n\u001b[1;32m    747\u001b[0m         \u001b[0;31m# Use NumPy API to support order\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    748\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcopy\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    749\u001b[0m             \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    750\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 751\u001b[0;31m             \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    752\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    753\u001b[0m         \u001b[0;31m# At this point array is a NumPy ndarray. We convert it to an array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m         \u001b[0;31m# container that is consistent with the input's namespace.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/pandas/core/generic.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(self, dtype, copy)\u001b[0m\n\u001b[1;32m   2149\u001b[0m     def __array__(\n\u001b[1;32m   2150\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mnpt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDTypeLike\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool_t\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2151\u001b[0m     ) -> np.ndarray:\n\u001b[1;32m   2152\u001b[0m         \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_values\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2153\u001b[0;31m         \u001b[0marr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2154\u001b[0m         if (\n\u001b[1;32m   2155\u001b[0m             \u001b[0mastype_is_view\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2156\u001b[0m             \u001b[0;32mand\u001b[0m \u001b[0musing_copy_on_write\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: could not convert string to float: 'Honda'"
     ]
    }
   ],
   "source": [
    "# Try to predict with random forest on price column (doesn't work)\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! We get a another `ValueError` (some of data is in string format rather than numerical format).\n",
    "\n",
    "```\n",
    "ValueError: could not convert string to float: 'Honda'\n",
    "```\n",
    "\n",
    "Machine learning models prefer to work with numbers than text. \n",
    "\n",
    "So we'll have to convert the non-numerical features into numbers first.\n",
    "\n",
    "The process of turning categorical features into numbers is often referred to as **encoding**.\n",
    "\n",
    "Scikit-Learn has a fantastic in-depth guide on [*Encoding categorical features*](https://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features).\n",
    "\n",
    "But let's look at one of the most straightforward ways to turn categorical features into numbers, one-hot encoding.\n",
    "\n",
    "In machine learning, [one-hot encoding](https://en.wikipedia.org/wiki/One-hot#Machine_learning_and_statistics) gives a value of `1` to the target value and a value of `0` to the other values.\n",
    "\n",
    "For example, let's say we had five samples and three car make options, Honda, Toyota, BMW.\n",
    "\n",
    "And our samples were:\n",
    "1. Honda\n",
    "2. BMW\n",
    "3. BMW\n",
    "4. Toyota\n",
    "5. Toyota\n",
    "\n",
    "If we were to one-hot encode these, it would look like:\n",
    "\n",
    "| Sample | Honda | Toyota | BMW |\n",
    "| ----- | ----- | ----- | ----- |\n",
    "| 1 | 1 | 0 | 0 |\n",
    "| 2 | 0 | 0 | 1 |\n",
    "| 3 | 0 | 0 | 1 |\n",
    "| 4 | 0 | 1 | 0 |\n",
    "| 5 | 0 | 1 | 0 |\n",
    "\n",
    "Notice how there's a 1 for each target value but a 0 for each other value.\n",
    "\n",
    "We can use the following steps to one-hot encode our dataset:\n",
    "1. Import [`sklearn.preprocessing.OneHotEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) to one-hot encode our features and [`sklearn.compose.ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) to target the specific columns of our DataFrame to transform.\n",
    "2. Define the categorical features we'd like to transform.\n",
    "3. Create an instance of the `OneHotEncoder`.\n",
    "4. Create an instance of `ColumnTransformer` and feed it the transforms we'd like to make.\n",
    "5. Fit the instance of the `ColumnTransformer` to our data and transform it with the [`fit_transform(X)`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer.fit_transform) method.\n",
    "\n",
    "> **Note:** In Scikit-Learn, the term \"transformer\" is often used to refer to something that *transforms* data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 3.54310e+04],\n",
       "       [1.00000e+00, 0.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        1.00000e+00, 1.92714e+05],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 8.47140e+04],\n",
       "       ...,\n",
       "       [0.00000e+00, 0.00000e+00, 1.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 6.66040e+04],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 2.15883e+05],\n",
       "       [0.00000e+00, 0.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 2.48360e+05]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. Import OneHotEncoder and ColumnTransformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "# 2. Define the categorical features to transform\n",
    "categorical_features = [\"Make\", \"Colour\", \"Doors\"]\n",
    "\n",
    "# 3. Create an instance of OneHotEncoder\n",
    "one_hot = OneHotEncoder()\n",
    "\n",
    "# 4. Create an instance of ColumnTransformer\n",
    "transformer = ColumnTransformer([(\"one_hot\", # name\n",
    "                                  one_hot, # transformer\n",
    "                                  categorical_features)], # columns to transform\n",
    "                                  remainder=\"passthrough\") # what to do with the rest of the columns? (\"passthrough\" = leave unchanged) \n",
    "\n",
    "# 5. Turn the categorical features into numbers (this will return an array-like sparse matrix, not a DataFrame)\n",
    "transformed_X = transformer.fit_transform(X)\n",
    "transformed_X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** You might be thinking why we considered `Doors` as a categorical variable. Which is a good question considering `Doors` is already numerical. Well, the answer is that `Doors` could be either numerical or categorical. However, I've decided to go with categorical, since where I'm from, number of doors is often a different *category* of car. For example, you can shop for 4-door cars or shop for 5-door cars (which always confused me since where's the 5th door?). However, you could experiment with treating this value as numerical or categorical, training a model on each, and then see how each model performs.\n",
    "\n",
    "Woah! Looks like our samples are all numerical, what did our data look like previously?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour  Odometer (KM)  Doors\n",
       "0   Honda  White          35431      4\n",
       "1     BMW   Blue         192714      5\n",
       "2   Honda  White          84714      4\n",
       "3  Toyota  White         154365      4\n",
       "4  Nissan   Blue         181577      3"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems `OneHotEncoder` and `ColumnTransformer` have turned all of our data samples into numbers.\n",
    "\n",
    "Let's check out the first transformed sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.0000e+00, 1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "       0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00, 0.0000e+00,\n",
       "       1.0000e+00, 0.0000e+00, 3.5431e+04])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View first transformed sample\n",
    "transformed_X[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And what were these values originally?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             Honda\n",
       "Colour           White\n",
       "Odometer (KM)    35431\n",
       "Doors                4\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View original first sample\n",
    "X.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.1 Nuemrically encoding data with pandas\n",
    "\n",
    "Another way we can numerically encode data is directly with pandas.\n",
    "\n",
    "We can use the [`pandas.get_dummies()`](https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html) (or `pd.get_dummies()` for short) method and then pass it our target columns.\n",
    "\n",
    "In return, we'll get a one-hot encoded version of our target columns.\n",
    "\n",
    "Let's remind ourselves of what our DataFrame looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431</td>\n",
       "      <td>4</td>\n",
       "      <td>15323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714</td>\n",
       "      <td>5</td>\n",
       "      <td>19943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714</td>\n",
       "      <td>4</td>\n",
       "      <td>28343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365</td>\n",
       "      <td>4</td>\n",
       "      <td>13434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577</td>\n",
       "      <td>3</td>\n",
       "      <td>14043</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour  Odometer (KM)  Doors  Price\n",
       "0   Honda  White          35431      4  15323\n",
       "1     BMW   Blue         192714      5  19943\n",
       "2   Honda  White          84714      4  28343\n",
       "3  Toyota  White         154365      4  13434\n",
       "4  Nissan   Blue         181577      3  14043"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View head of original DataFrame\n",
    "car_sales.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful, now let's use `pd.get_dummies()` to turn our categorical variables into one-hot encoded variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Doors</th>\n",
       "      <th>Make_BMW</th>\n",
       "      <th>Make_Honda</th>\n",
       "      <th>Make_Nissan</th>\n",
       "      <th>Make_Toyota</th>\n",
       "      <th>Colour_Black</th>\n",
       "      <th>Colour_Blue</th>\n",
       "      <th>Colour_Green</th>\n",
       "      <th>Colour_Red</th>\n",
       "      <th>Colour_White</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Doors  Make_BMW  Make_Honda  Make_Nissan  Make_Toyota  Colour_Black  \\\n",
       "0        4     False        True        False        False         False   \n",
       "1        5      True       False        False        False         False   \n",
       "2        4     False        True        False        False         False   \n",
       "3        4     False       False        False         True         False   \n",
       "4        3     False       False         True        False         False   \n",
       "..     ...       ...         ...          ...          ...           ...   \n",
       "995      4     False       False        False         True          True   \n",
       "996      3     False       False         True        False         False   \n",
       "997      4     False       False         True        False         False   \n",
       "998      4     False        True        False        False         False   \n",
       "999      4     False       False        False         True         False   \n",
       "\n",
       "     Colour_Blue  Colour_Green  Colour_Red  Colour_White  \n",
       "0          False         False       False          True  \n",
       "1           True         False       False         False  \n",
       "2          False         False       False          True  \n",
       "3          False         False       False          True  \n",
       "4           True         False       False         False  \n",
       "..           ...           ...         ...           ...  \n",
       "995        False         False       False         False  \n",
       "996        False         False       False          True  \n",
       "997         True         False       False         False  \n",
       "998        False         False       False          True  \n",
       "999         True         False       False         False  \n",
       "\n",
       "[1000 rows x 10 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One-hot encode categorical variables\n",
    "categorical_variables = [\"Make\", \"Colour\", \"Doors\"]\n",
    "dummies = pd.get_dummies(data=car_sales[categorical_variables])\n",
    "dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Notice how there's a new column for each categorical option (e.g. `Make_BMW`, `Make_Honda`, etc).\n",
    "\n",
    "But also notice how it also missed the `Doors` column?\n",
    "\n",
    "This is because `Doors` is already numeric, so for `pd.get_dummies()` to work on it, we can change it to type `object`.\n",
    "\n",
    "By default, `pd.get_dummies()` also turns all of the values to bools (`True` or `False`).\n",
    "\n",
    "We can get the returned values as `0` or `1` by setting `dtype=float`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make_BMW</th>\n",
       "      <th>Make_Honda</th>\n",
       "      <th>Make_Nissan</th>\n",
       "      <th>Make_Toyota</th>\n",
       "      <th>Colour_Black</th>\n",
       "      <th>Colour_Blue</th>\n",
       "      <th>Colour_Green</th>\n",
       "      <th>Colour_Red</th>\n",
       "      <th>Colour_White</th>\n",
       "      <th>Doors_3</th>\n",
       "      <th>Doors_4</th>\n",
       "      <th>Doors_5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make_BMW  Make_Honda  Make_Nissan  Make_Toyota  Colour_Black  \\\n",
       "0         0.0         1.0          0.0          0.0           0.0   \n",
       "1         1.0         0.0          0.0          0.0           0.0   \n",
       "2         0.0         1.0          0.0          0.0           0.0   \n",
       "3         0.0         0.0          0.0          1.0           0.0   \n",
       "4         0.0         0.0          1.0          0.0           0.0   \n",
       "..        ...         ...          ...          ...           ...   \n",
       "995       0.0         0.0          0.0          1.0           1.0   \n",
       "996       0.0         0.0          1.0          0.0           0.0   \n",
       "997       0.0         0.0          1.0          0.0           0.0   \n",
       "998       0.0         1.0          0.0          0.0           0.0   \n",
       "999       0.0         0.0          0.0          1.0           0.0   \n",
       "\n",
       "     Colour_Blue  Colour_Green  Colour_Red  Colour_White  Doors_3  Doors_4  \\\n",
       "0            0.0           0.0         0.0           1.0      0.0      1.0   \n",
       "1            1.0           0.0         0.0           0.0      0.0      0.0   \n",
       "2            0.0           0.0         0.0           1.0      0.0      1.0   \n",
       "3            0.0           0.0         0.0           1.0      0.0      1.0   \n",
       "4            1.0           0.0         0.0           0.0      1.0      0.0   \n",
       "..           ...           ...         ...           ...      ...      ...   \n",
       "995          0.0           0.0         0.0           0.0      0.0      1.0   \n",
       "996          0.0           0.0         0.0           1.0      1.0      0.0   \n",
       "997          1.0           0.0         0.0           0.0      0.0      1.0   \n",
       "998          0.0           0.0         0.0           1.0      0.0      1.0   \n",
       "999          1.0           0.0         0.0           0.0      0.0      1.0   \n",
       "\n",
       "     Doors_5  \n",
       "0        0.0  \n",
       "1        1.0  \n",
       "2        0.0  \n",
       "3        0.0  \n",
       "4        0.0  \n",
       "..       ...  \n",
       "995      0.0  \n",
       "996      0.0  \n",
       "997      0.0  \n",
       "998      0.0  \n",
       "999      0.0  \n",
       "\n",
       "[1000 rows x 12 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Have to convert doors to object for dummies to work on it...\n",
    "car_sales[\"Doors\"] = car_sales[\"Doors\"].astype(object)\n",
    "dummies = pd.get_dummies(data=car_sales[[\"Make\", \"Colour\", \"Doors\"]],\n",
    "                         dtype=float)\n",
    "dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo!\n",
    "\n",
    "We've now turned our data into fully numeric form using Scikit-Learn and pandas.\n",
    "\n",
    "Now you might be wondering...\n",
    "\n",
    "**Should you use Scikit-Learn or pandas for turning data into numerical form?**\n",
    "\n",
    "And the answer is either.\n",
    "\n",
    "But as a rule of thumb:\n",
    "* If you're performing **quick data analysis and running small modelling experiments**, use `pandas` as it's generally quite fast to get up and running.\n",
    "* If you're performing a **larger scale modelling experiment** or would like to put your **data processing steps into a production pipeline**, I'd recommend leaning towards Scikit-Learn, specifically a [Scikit-Learn Pipeline](https://scikit-learn.org/stable/modules/compose.html#pipeline) (chaining together multiple estimator/modelling steps).\n",
    "\n",
    "Since we've turned our data into numerical form, how about we try and fit our model again?\n",
    "\n",
    "Let's recreate a train/test split except this time we'll use `transformed_X` instead of `X`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3235867221569877"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "# Create train and test splits with transformed_X\n",
    "X_train, X_test, y_train, y_test = train_test_split(transformed_X,\n",
    "                                                    y,\n",
    "                                                    test_size=0.2)\n",
    "\n",
    "# Create the model instance\n",
    "model = RandomForestRegressor()\n",
    "\n",
    "# Fit the model on the numerical data (this errored before since our data wasn't fully numeric)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Score the model (returns r^2 metric by default, also called coefficient of determination, higher is better)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 What if there were missing values in the data?\n",
    "\n",
    "Holes in the data means holes in the patterns your machine learning model can learn.\n",
    "\n",
    "Many machine learning models don't work well or produce errors when they're used on datasets with missing values.\n",
    "\n",
    "A missing value can appear as a blank, as a `NaN` or something similar.\n",
    "\n",
    "There are two main options when dealing with missing values:\n",
    "\n",
    "1. **Fill them with some given or calculated value (imputation)** - For example, you might fill missing values of a numerical column with the mean of all the other values. The practice of calculating or figuring out how to fill missing values in a dataset is called **imputing**. For a great resource on imputing missing values, I'd recommend refering to the [Scikit-Learn user guide](https://scikit-learn.org/stable/modules/impute.html).\n",
    "2. **Remove them** - If a row or sample has missing values, you may opt to remove them from your dataset completely. However, this potentially results in using less data to build your model.\n",
    "\n",
    "> **Note:** Dealing with missing values differs from problem to problem, meaning there's no 100% best way to fill missing values across datasets and problem types. It will often take careful experimentation and practice to figure out the best way to deal with missing values in your own datasets.\n",
    "\n",
    "To practice dealing with missing values, let's import a version of the `car_sales` dataset with several missing values (namely [`car-sales-extended-missing-data.csv`](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended-missing-data.csv))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15323.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>19943.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28343.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>13434.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>14043.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Honda</td>\n",
       "      <td>Red</td>\n",
       "      <td>42652.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>23883.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>Blue</td>\n",
       "      <td>163453.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8473.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20306.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>White</td>\n",
       "      <td>130538.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9374.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Honda</td>\n",
       "      <td>Blue</td>\n",
       "      <td>51029.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>26683.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour  Odometer (KM)  Doors    Price\n",
       "0   Honda  White        35431.0    4.0  15323.0\n",
       "1     BMW   Blue       192714.0    5.0  19943.0\n",
       "2   Honda  White        84714.0    4.0  28343.0\n",
       "3  Toyota  White       154365.0    4.0  13434.0\n",
       "4  Nissan   Blue       181577.0    3.0  14043.0\n",
       "5   Honda    Red        42652.0    4.0  23883.0\n",
       "6  Toyota   Blue       163453.0    4.0   8473.0\n",
       "7   Honda  White            NaN    4.0  20306.0\n",
       "8     NaN  White       130538.0    4.0   9374.0\n",
       "9   Honda   Blue        51029.0    4.0  26683.0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import car sales dataframe with missing values\n",
    "# car_sales_missing = pd.read_csv(\"../data/car-sales-extended-missing-data.csv\") # load from local directory\n",
    "car_sales_missing = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv\") # read directly from URL (source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended-missing-data.csv)\n",
    "car_sales_missing.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the `NaN` value in row 7 for the `Odometer (KM)` column, that means pandas has detected a missing value there.\n",
    "\n",
    "However, if you're dataset is large, it's likely you aren't going to go through it sample by sample to find the missing values.\n",
    "\n",
    "Luckily, pandas has a method called [`pd.DataFrame.isna()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isna.html) which is able to detect missing values.\n",
    "\n",
    "Let's try it on our DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             49\n",
       "Colour           50\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the sum of all missing values\n",
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm... seems there's about 50 or so missing values per column.\n",
    "\n",
    "How about we try and split the data into features and labels, then convert the categorical data to numbers, then split the data into training and test and then try and fit a model on it (just like we did before)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing X values:\n",
      "Make             49\n",
      "Colour           50\n",
      "Odometer (KM)    50\n",
      "Doors            50\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Create features\n",
    "X_missing = car_sales_missing.drop(\"Price\", axis=1)\n",
    "print(f\"Number of missing X values:\\n{X_missing.isna().sum()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing y values: 50\n"
     ]
    }
   ],
   "source": [
    "# Create labels\n",
    "y_missing = car_sales_missing[\"Price\"]\n",
    "print(f\"Number of missing y values: {y_missing.isna().sum()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can convert the categorical columns into one-hot encodings (just as before)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        0.00000e+00, 3.54310e+04],\n",
       "       [1.00000e+00, 0.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 1.92714e+05],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        0.00000e+00, 8.47140e+04],\n",
       "       ...,\n",
       "       [0.00000e+00, 0.00000e+00, 1.00000e+00, ..., 0.00000e+00,\n",
       "        0.00000e+00, 6.66040e+04],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        0.00000e+00, 2.15883e+05],\n",
       "       [0.00000e+00, 0.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        0.00000e+00, 2.48360e+05]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's convert the categorical columns to one hot encoded (code copied from above)\n",
    "# Turn the categories (Make and Colour) into numbers\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "categorical_features = [\"Make\", \"Colour\", \"Doors\"]\n",
    "\n",
    "one_hot = OneHotEncoder()\n",
    "\n",
    "transformer = ColumnTransformer([(\"one_hot\", \n",
    "                                  one_hot, \n",
    "                                  categorical_features)],\n",
    "                                remainder=\"passthrough\",\n",
    "                                sparse_threshold=0) # return a sparse matrix or not\n",
    "\n",
    "transformed_X_missing = transformer.fit_transform(X_missing)\n",
    "transformed_X_missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's split the missing data samples into train and test sets and then try to fit and score a model on them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Input y contains NaN.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[52], line 8\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;66;03m# Fit and score a model\u001b[39;00m\n\u001b[1;32m      7\u001b[0m model \u001b[38;5;241m=\u001b[39m RandomForestRegressor()\n\u001b[0;32m----> 8\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my_train\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      9\u001b[0m model\u001b[38;5;241m.\u001b[39mscore(X_test, y_test)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py:1473\u001b[0m, in \u001b[0;36m_fit_context.<locals>.decorator.<locals>.wrapper\u001b[0;34m(estimator, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1466\u001b[0m     estimator\u001b[38;5;241m.\u001b[39m_validate_params()\n\u001b[1;32m   1468\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m config_context(\n\u001b[1;32m   1469\u001b[0m     skip_parameter_validation\u001b[38;5;241m=\u001b[39m(\n\u001b[1;32m   1470\u001b[0m         prefer_skip_nested_validation \u001b[38;5;129;01mor\u001b[39;00m global_skip_validation\n\u001b[1;32m   1471\u001b[0m     )\n\u001b[1;32m   1472\u001b[0m ):\n\u001b[0;32m-> 1473\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfit_method\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py:363\u001b[0m, in \u001b[0;36mBaseForest.fit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    360\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m issparse(y):\n\u001b[1;32m    361\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msparse multilabel-indicator for y is not supported.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m--> 363\u001b[0m X, y \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_validate_data\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    364\u001b[0m \u001b[43m    \u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    365\u001b[0m \u001b[43m    \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    366\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmulti_output\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    367\u001b[0m \u001b[43m    \u001b[49m\u001b[43maccept_sparse\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcsc\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    368\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mDTYPE\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    369\u001b[0m \u001b[43m    \u001b[49m\u001b[43mforce_all_finite\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    370\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    371\u001b[0m \u001b[38;5;66;03m# _compute_missing_values_in_feature_mask checks if X has missing values and\u001b[39;00m\n\u001b[1;32m    372\u001b[0m \u001b[38;5;66;03m# will raise an error if the underlying tree base estimator can't handle missing\u001b[39;00m\n\u001b[1;32m    373\u001b[0m \u001b[38;5;66;03m# values. Only the criterion is required to determine if the tree supports\u001b[39;00m\n\u001b[1;32m    374\u001b[0m \u001b[38;5;66;03m# missing values.\u001b[39;00m\n\u001b[1;32m    375\u001b[0m estimator \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mestimator)(criterion\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcriterion)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py:650\u001b[0m, in \u001b[0;36mBaseEstimator._validate_data\u001b[0;34m(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)\u001b[0m\n\u001b[1;32m    648\u001b[0m         y \u001b[38;5;241m=\u001b[39m check_array(y, input_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124my\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mcheck_y_params)\n\u001b[1;32m    649\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 650\u001b[0m         X, y \u001b[38;5;241m=\u001b[39m \u001b[43mcheck_X_y\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mcheck_params\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    651\u001b[0m     out \u001b[38;5;241m=\u001b[39m X, y\n\u001b[1;32m    653\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m no_val_X \u001b[38;5;129;01mand\u001b[39;00m check_params\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mensure_2d\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mTrue\u001b[39;00m):\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:1318\u001b[0m, in \u001b[0;36mcheck_X_y\u001b[0;34m(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1297\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1298\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mestimator_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m requires y to be passed, but the target y is None\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1299\u001b[0m     )\n\u001b[1;32m   1301\u001b[0m X \u001b[38;5;241m=\u001b[39m check_array(\n\u001b[1;32m   1302\u001b[0m     X,\n\u001b[1;32m   1303\u001b[0m     accept_sparse\u001b[38;5;241m=\u001b[39maccept_sparse,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1315\u001b[0m     input_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mX\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m   1316\u001b[0m )\n\u001b[0;32m-> 1318\u001b[0m y \u001b[38;5;241m=\u001b[39m \u001b[43m_check_y\u001b[49m\u001b[43m(\u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmulti_output\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmulti_output\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my_numeric\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43my_numeric\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mestimator\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1320\u001b[0m check_consistent_length(X, y)\n\u001b[1;32m   1322\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m X, y\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:1328\u001b[0m, in \u001b[0;36m_check_y\u001b[0;34m(y, multi_output, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1326\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Isolated part of check_X_y dedicated to y validation\"\"\"\u001b[39;00m\n\u001b[1;32m   1327\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m multi_output:\n\u001b[0;32m-> 1328\u001b[0m     y \u001b[38;5;241m=\u001b[39m \u001b[43mcheck_array\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1329\u001b[0m \u001b[43m        \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1330\u001b[0m \u001b[43m        \u001b[49m\u001b[43maccept_sparse\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcsr\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1331\u001b[0m \u001b[43m        \u001b[49m\u001b[43mforce_all_finite\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m   1332\u001b[0m \u001b[43m        \u001b[49m\u001b[43mensure_2d\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m   1333\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m   1334\u001b[0m \u001b[43m        \u001b[49m\u001b[43minput_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43my\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1335\u001b[0m \u001b[43m        \u001b[49m\u001b[43mestimator\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mestimator\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1336\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1337\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1338\u001b[0m     estimator_name \u001b[38;5;241m=\u001b[39m _check_estimator_name(estimator)\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:1064\u001b[0m, in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001b[0m\n\u001b[1;32m   1058\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1059\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFound array with dim \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m. \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m expected <= 2.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1060\u001b[0m         \u001b[38;5;241m%\u001b[39m (array\u001b[38;5;241m.\u001b[39mndim, estimator_name)\n\u001b[1;32m   1061\u001b[0m     )\n\u001b[1;32m   1063\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m force_all_finite:\n\u001b[0;32m-> 1064\u001b[0m     \u001b[43m_assert_all_finite\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1065\u001b[0m \u001b[43m        \u001b[49m\u001b[43marray\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1066\u001b[0m \u001b[43m        \u001b[49m\u001b[43minput_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minput_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1067\u001b[0m \u001b[43m        \u001b[49m\u001b[43mestimator_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mestimator_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1068\u001b[0m \u001b[43m        \u001b[49m\u001b[43mallow_nan\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_all_finite\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mallow-nan\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1069\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1071\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m copy:\n\u001b[1;32m   1072\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m _is_numpy_namespace(xp):\n\u001b[1;32m   1073\u001b[0m         \u001b[38;5;66;03m# only make a copy if `array` and `array_orig` may share memory`\u001b[39;00m\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:123\u001b[0m, in \u001b[0;36m_assert_all_finite\u001b[0;34m(X, allow_nan, msg_dtype, estimator_name, input_name)\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m first_pass_isfinite:\n\u001b[1;32m    121\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[0;32m--> 123\u001b[0m \u001b[43m_assert_all_finite_element_wise\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    124\u001b[0m \u001b[43m    \u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    125\u001b[0m \u001b[43m    \u001b[49m\u001b[43mxp\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mxp\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    126\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallow_nan\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mallow_nan\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    127\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmsg_dtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmsg_dtype\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    128\u001b[0m \u001b[43m    \u001b[49m\u001b[43mestimator_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mestimator_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    129\u001b[0m \u001b[43m    \u001b[49m\u001b[43minput_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minput_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    130\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py:172\u001b[0m, in \u001b[0;36m_assert_all_finite_element_wise\u001b[0;34m(X, xp, allow_nan, msg_dtype, estimator_name, input_name)\u001b[0m\n\u001b[1;32m    155\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m estimator_name \u001b[38;5;129;01mand\u001b[39;00m input_name \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mX\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m has_nan_error:\n\u001b[1;32m    156\u001b[0m     \u001b[38;5;66;03m# Improve the error message on how to handle missing values in\u001b[39;00m\n\u001b[1;32m    157\u001b[0m     \u001b[38;5;66;03m# scikit-learn.\u001b[39;00m\n\u001b[1;32m    158\u001b[0m     msg_err \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m    159\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00mestimator_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m does not accept missing values\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    160\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m encoded as NaN natively. For supervised learning, you might want\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#estimators-that-handle-nan-values\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    171\u001b[0m     )\n\u001b[0;32m--> 172\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(msg_err)\n",
      "\u001b[0;31mValueError\u001b[0m: Input y contains NaN."
     ]
    }
   ],
   "source": [
    "# Split data into training and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(transformed_X_missing,\n",
    "                                                    y_missing,\n",
    "                                                    test_size=0.2)\n",
    "\n",
    "# Fit and score a model\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahh... dam! Another `ValueError` (our input data contains missing values).\n",
    "\n",
    "`ValueError: Input y contains NaN.`\n",
    "\n",
    "Looks like the model we're trying to use doesn't work with missing values.\n",
    "\n",
    "When we try to fit it on a dataset with missing samples, Scikit-Learn produces an error similar to:\n",
    "\n",
    "`ValueError: Input X contains NaN. RandomForestRegressor does not accept missing values encoded as NaN natively...`\n",
    "\n",
    "Looks like if we want to use `RandomForestRegressor`, we'll have to either fill or remove the missing values.\n",
    "\n",
    "<details class=\"tip\">\n",
    "    <summary><strong>Note:</strong> Scikit-Learn does have a \n",
    "        <a href=\"https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values\">list of models which can handle NaNs or missing values directly</a>.\n",
    "    </summary>\n",
    "    <p>Such as, \n",
    "        <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html\"><code>sklearn.ensemble.HistGradientBoostingClassifier</code></a> \n",
    "        or \n",
    "        <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html\"><code>sklearn.ensemble.HistGradientBoostingRegressor</code></a>.\n",
    "    </p>\n",
    "    <p>As an experiment, you may want to try the following:</p>\n",
    "    <pre><code>\n",
    "from sklearn.ensemble import HistGradientBoostingRegressor\n",
    "\n",
    "\\# Try a model that can handle NaNs natively\n",
    "nan_model = HistGradientBoostingRegressor()\n",
    "nan_model.fit(X_train, y_train)\n",
    "nan_model.score(X_test, y_test)\n",
    "    </code></pre>\n",
    "</details>\n",
    "Let's see what values are missing again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             49\n",
       "Colour           50\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can fill (impute) or remove these?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.1 Fill missing data with pandas\n",
    "\n",
    "Let's see how we might fill missing values with pandas.\n",
    "\n",
    "For categorical values, one of the simplest ways is to fill the missing fields with the string `\"missing\"`.\n",
    "\n",
    "We could do this for the `Make` and `Colour` features.\n",
    "\n",
    "As for the `Doors` feature, we could use `\"missing\"` or we could fill it with the most common option of `4`.\n",
    "\n",
    "With the `Odometer (KM)` feature, we can use the mean value of all the other values in the column.\n",
    "\n",
    "And finally, for those samples which are missing a `Price` value, we can remove them (since `Price` is the target value, removing probably causes less harm than imputing, however, you could design an experiment to test this).\n",
    "\n",
    "In summary:\n",
    "\n",
    "| Column/Feature | Fill missing value with | \n",
    "| ----- | ----- |\n",
    "| `Make` | `\"missing\"` |\n",
    "| `Colour` | `\"missing\"` |\n",
    "| `Doors` | 4 (most common value) |\n",
    "| `Odometer (KM)` | mean of `Odometer (KM)` | \n",
    "| `Price` (target) | NA, remove samples missing `Price` |\n",
    "\n",
    "\n",
    "> **Note:** The practice of filling missing data with given or calculated values is called [**imputation**](https://scikit-learn.org/stable/modules/impute.html). And it's important to remember there's no perfect way to fill missing data (unless it's with data that should've actually been there in the first place). The methods we're using are only one of many. The techniques you use will depend heavily on your dataset. A good place to look would be searching for \"data imputation techniques\".\n",
    "\n",
    "Let's start with the `Make` column.\n",
    "\n",
    "We can use the pandas method [`fillna(value=\"missing\", inplace=True)`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html) to fill all the missing values with the string `\"missing\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill the missing values in the Make column\n",
    "# Note: In previous versions of pandas, inplace=True was possible, however this will be changed in a future version, can use reassignment instead.\n",
    "# car_sales_missing[\"Make\"].fillna(value=\"missing\", inplace=True)\n",
    "\n",
    "car_sales_missing[\"Make\"] = car_sales_missing[\"Make\"].fillna(value=\"missing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can do the same with the `Colour` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: In previous versions of pandas, inplace=True was possible, however this will be changed in a future version, can use reassignment instead.\n",
    "# car_sales_missing[\"Colour\"].fillna(value=\"missing\", inplace=True)\n",
    "\n",
    "# Fill the Colour column\n",
    "car_sales_missing[\"Colour\"] = car_sales_missing[\"Colour\"].fillna(value=\"missing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many missing values do we have now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              0\n",
       "Colour            0\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful! We're making some progress.\n",
    "\n",
    "Now let's fill the `Doors` column with `4` (the most common value), this is the same as filling it with the [median](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.median.html) or [mode](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mode.html) of the `Doors` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Doors\n",
       "4.0    811\n",
       "5.0     75\n",
       "3.0     64\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the most common value of the Doors column\n",
    "car_sales_missing[\"Doors\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill the Doors column with the most common value\n",
    "car_sales_missing[\"Doors\"] = car_sales_missing[\"Doors\"].fillna(value=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll fill the `Odometer (KM)` column with the mean value of itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill the Odometer (KM) column\n",
    "# Old: car_sales_missing[\"Odometer (KM)\"].fillna(value=car_sales_missing[\"Odometer (KM)\"].mean(), inplace=True)\n",
    "\n",
    "car_sales_missing[\"Odometer (KM)\"] = car_sales_missing[\"Odometer (KM)\"].fillna(value=car_sales_missing[\"Odometer (KM)\"].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many missing values do we have now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              0\n",
       "Colour            0\n",
       "Odometer (KM)     0\n",
       "Doors             0\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of missing values\n",
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo! That's looking a lot better.\n",
    "\n",
    "Finally, we can remove the rows which are missing the target value `Price`.\n",
    "\n",
    "> **Note:** Another option would be to impute the `Price` value with the mean or median or some other calculated value (such as by using similar cars to estimate the price), however, to keep things simple and prevent introducing too many fake labels to the data, we'll remove the samples missing a `Price` value. \n",
    "\n",
    "We can remove rows with missing values in place from a pandas DataFrame with the [`pandas.DataFrame.dropna(inplace=True)`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.dropna.html) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove rows with missing Price labels\n",
    "car_sales_missing.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That should be no more missing values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             0\n",
       "Colour           0\n",
       "Odometer (KM)    0\n",
       "Doors            0\n",
       "Price            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of missing values\n",
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we removed samples missing a `Price` value, there's now less overall samples but none of them have missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "950"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of total samples (previously was 1000)\n",
    "len(car_sales_missing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we fit a model now?\n",
    "\n",
    "Let's try!\n",
    "\n",
    "First we'll create the features and labels.\n",
    "\n",
    "Then we'll convert categorical variables into numbers via one-hot encoding.\n",
    "\n",
    "Then we'll split the data into training and test sets just like before.\n",
    "\n",
    "Finally, we'll try to fit a `RandomForestRegressor()` model to the newly filled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing X values:\n",
      "Make             0\n",
      "Colour           0\n",
      "Odometer (KM)    0\n",
      "Doors            0\n",
      "dtype: int64\n",
      "Number of missing y values: 0\n"
     ]
    }
   ],
   "source": [
    "# Create features\n",
    "X_missing = car_sales_missing.drop(\"Price\", axis=1)\n",
    "print(f\"Number of missing X values:\\n{X_missing.isna().sum()}\")\n",
    "\n",
    "# Create labels\n",
    "y_missing = car_sales_missing[\"Price\"]\n",
    "print(f\"Number of missing y values: {y_missing.isna().sum()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 3.54310e+04],\n",
       "       [1.00000e+00, 0.00000e+00, 0.00000e+00, ..., 0.00000e+00,\n",
       "        1.00000e+00, 1.92714e+05],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 8.47140e+04],\n",
       "       ...,\n",
       "       [0.00000e+00, 0.00000e+00, 1.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 6.66040e+04],\n",
       "       [0.00000e+00, 1.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 2.15883e+05],\n",
       "       [0.00000e+00, 0.00000e+00, 0.00000e+00, ..., 1.00000e+00,\n",
       "        0.00000e+00, 2.48360e+05]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "categorical_features = [\"Make\", \"Colour\", \"Doors\"]\n",
    "\n",
    "one_hot = OneHotEncoder()\n",
    "\n",
    "transformer = ColumnTransformer([(\"one_hot\", \n",
    "                                  one_hot, \n",
    "                                  categorical_features)],\n",
    "                                remainder=\"passthrough\",\n",
    "                                sparse_threshold=0) # return a sparse matrix or not\n",
    "\n",
    "transformed_X_missing = transformer.fit_transform(X_missing)\n",
    "transformed_X_missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22011714008302485"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split data into training and test sets\n",
    "np.random.seed(42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(transformed_X_missing,\n",
    "                                                    y_missing,\n",
    "                                                    test_size=0.2)\n",
    "\n",
    "# Fit and score a model\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic!!!\n",
    "\n",
    "Looks like filling the missing values with pandas worked!\n",
    "\n",
    "Our model can be fit to the data without issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.2 Filling missing data and transforming categorical data with Scikit-Learn\n",
    "\n",
    "Now we've filled the missing columns using pandas functions, you might be thinking, \"Why pandas? I thought this was a Scikit-Learn introduction?\".\n",
    "\n",
    "Not to worry, Scikit-Learn provides a class called [`sklearn.impute.SimpleImputer()`](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html) which allows us to do a similar thing.\n",
    "\n",
    "`SimpleImputer()` transforms data by filling missing values with a given `strategy` parameter.\n",
    "\n",
    "And we can use it to fill the missing values in our DataFrame as above.\n",
    "\n",
    "At the moment, our dataframe has no mising values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             0\n",
       "Colour           0\n",
       "Odometer (KM)    0\n",
       "Doors            0\n",
       "Price            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reimport it so it has missing values and we can fill them with Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             49\n",
       "Colour           50\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reimport the DataFrame (so that all the missing values are back)\n",
    "# car_sales_missing = pd.read_csv(\"../data/car-sales-extended-missing-data.csv\") # read from local directory\n",
    "car_sales_missing = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv\") # read directly from URL (source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended-missing-data.csv)\n",
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin, we'll remove the rows which are missing a `Price` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the rows with missing in the Price column\n",
    "car_sales_missing.dropna(subset=[\"Price\"], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there are no rows missing a `Price` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             47\n",
       "Colour           46\n",
       "Odometer (KM)    48\n",
       "Doors            47\n",
       "Price             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we don't have to fill any `Price` values, let's split our data into features (`X`) and labels (`y`).\n",
    "\n",
    "We'll also split the data into training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split into X and y\n",
    "X = car_sales_missing.drop(\"Price\", axis=1)\n",
    "y = car_sales_missing[\"Price\"]\n",
    "\n",
    "# Split data into train and test\n",
    "np.random.seed(42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,\n",
    "                                                    y,\n",
    "                                                    test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** We've split the data into train & test sets here first to perform filling missing values on them separately. This is best practice as the test set is supposed to emulate data the model has never seen before. For categorical variables, it's generally okay to fill values across the whole dataset. However, for numerical vairables, you should **only fill values on the test set that have been computed from the training set**.\n",
    "\n",
    "Training and test sets created!\n",
    "\n",
    "Let's now setup a few instances of `SimpleImputer()` to fill various missing values.\n",
    "\n",
    "We'll use the following strategies and fill values:\n",
    "* For categorical columns (`Make`, `Colour`), `strategy=\"constant\"`, `fill_value=\"missing\"` (fill the missing samples with a consistent value of `\"missing\"`.\n",
    "* For the `Door` column, `strategy=\"constant\"`, `fill_value=4` (fill the missing samples with a consistent value of `4`).\n",
    "* For the numerical column (`Odometer (KM)`), `strategy=\"mean\"` (fill the missing samples with the mean of the target column).\n",
    "  * **Note:** There are more `strategy` and fill options in the [`SimpleImputer()` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# Create categorical variable imputer\n",
    "cat_imputer = SimpleImputer(strategy=\"constant\", fill_value=\"missing\")\n",
    "\n",
    "# Create Door column imputer\n",
    "door_imputer = SimpleImputer(strategy=\"constant\", fill_value=4)\n",
    "\n",
    "# Create Odometer (KM) column imputer\n",
    "num_imputer = SimpleImputer(strategy=\"mean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imputers created!\n",
    "\n",
    "Now let's define which columns we'd like to impute on.\n",
    "\n",
    "Why?\n",
    "\n",
    "Because we'll need these shortly (I'll explain in the next text cell)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define different column features\n",
    "categorical_features = [\"Make\", \"Colour\"]\n",
    "door_feature = [\"Doors\"]\n",
    "numerical_feature = [\"Odometer (KM)\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns defined!\n",
    "\n",
    "Now how might we transform our columns?\n",
    "\n",
    "Hint: we can use the [`sklearn.compose.ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) class from Scikit-Learn, in a similar way to what we did before to get our data to all numeric values.\n",
    "\n",
    "That's the reason we defined the columns we'd like to transform.\n",
    "\n",
    "So we can use the `ColumnTransformer()` class.\n",
    "\n",
    "`ColumnTransformer()` takes as input a list of tuples in the form `(name_of_transform, transformer_to_use, columns_to_transform)` specifying which columns to transform and how to transform them.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "imputer = ColumnTransformer([\n",
    "    (\"cat_imputer\", cat_imputer, categorical_features)\n",
    "])\n",
    "```\n",
    "\n",
    "In this case, the variables in the tuple are:\n",
    "* `name_of_transform` = `\"cat_imputer\"`\n",
    "* `transformer_to_use` = `cat_imputer` (the instance of `SimpleImputer()` we defined above)\n",
    "* `columns_to_transform` = `categorical_features` (the list of categorical features we defined above).\n",
    "\n",
    "Let's exapnd upon this by extending the example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "# Create series of column transforms to perform\n",
    "imputer = ColumnTransformer([\n",
    "    (\"cat_imputer\", cat_imputer, categorical_features),\n",
    "    (\"door_imputer\", door_imputer, door_feature),\n",
    "    (\"num_imputer\", num_imputer, numerical_feature)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "The next step is to fit our `ColumnTransformer()` instance (`imputer`) to the training data and transform the testing data.\n",
    "\n",
    "In other words we want to:\n",
    "1. Learn the imputation values from the training set.\n",
    "2. Fill the missing values in the training set with the values learned in 1. \n",
    "3. Fill the missing values in the testing set with the values learned in 1.\n",
    "\n",
    "Why this way?\n",
    "\n",
    "In our case, we're not calculating many variables (except the mean of the `Odometer (KM)` column), however, remember that the test set should always remain as unseen data.\n",
    "\n",
    "So **when filling values in the test set, they should only be with values calculated or imputed from the training set**s.\n",
    "\n",
    "We can achieve steps 1 & 2 simultaneously with the [`ColumnTransformer.fit_transform()`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer.fit_transform) method (`fit` = find the values to fill, `transform` = fill them).\n",
    "\n",
    "And then we can perform step 3 with the [`ColumnTransformer.transform()`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer.transform) method (we only want to transform the test set, not learn different values to fill)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['Honda', 'White', 4.0, 71934.0],\n",
       "       ['Toyota', 'Red', 4.0, 162665.0],\n",
       "       ['Honda', 'White', 4.0, 42844.0],\n",
       "       ...,\n",
       "       ['Toyota', 'White', 4.0, 196225.0],\n",
       "       ['Honda', 'Blue', 4.0, 133117.0],\n",
       "       ['Honda', 'missing', 4.0, 150582.0]], dtype=object)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find values to fill and transform training data\n",
    "filled_X_train = imputer.fit_transform(X_train)\n",
    "\n",
    "# Fill values in to the test set with values learned from the training set\n",
    "filled_X_test = imputer.transform(X_test)\n",
    "\n",
    "# Check filled X_train\n",
    "filled_X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful!\n",
    "\n",
    "Let's now turn our `filled_X_train` and `filled_X_test` arrays into DataFrames to inspect their missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             0\n",
       "Colour           0\n",
       "Doors            0\n",
       "Odometer (KM)    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get our transformed data array's back into DataFrame's\n",
    "filled_X_train_df = pd.DataFrame(filled_X_train, \n",
    "                                 columns=[\"Make\", \"Colour\", \"Doors\", \"Odometer (KM)\"])\n",
    "\n",
    "filled_X_test_df = pd.DataFrame(filled_X_test, \n",
    "                                columns=[\"Make\", \"Colour\", \"Doors\", \"Odometer (KM)\"])\n",
    "\n",
    "# Check missing data in training set\n",
    "filled_X_train_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And is there any missing data in the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             0\n",
       "Colour           0\n",
       "Doors            0\n",
       "Odometer (KM)    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check missing data in the testing set\n",
    "filled_X_test_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about the original?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             47\n",
       "Colour           46\n",
       "Odometer (KM)    48\n",
       "Doors            47\n",
       "Price             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to see the original... still missing values\n",
    "car_sales_missing.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect!\n",
    "\n",
    "No more missing values! \n",
    "\n",
    "But wait...\n",
    "\n",
    "Is our data all numerical?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>4.0</td>\n",
       "      <td>71934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>Red</td>\n",
       "      <td>4.0</td>\n",
       "      <td>162665.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>4.0</td>\n",
       "      <td>42844.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>4.0</td>\n",
       "      <td>195829.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Honda</td>\n",
       "      <td>Blue</td>\n",
       "      <td>4.0</td>\n",
       "      <td>219217.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour Doors Odometer (KM)\n",
       "0   Honda  White   4.0       71934.0\n",
       "1  Toyota    Red   4.0      162665.0\n",
       "2   Honda  White   4.0       42844.0\n",
       "3   Honda  White   4.0      195829.0\n",
       "4   Honda   Blue   4.0      219217.0"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filled_X_train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahh... looks like our `Make` and `Colour` columns are still strings.\n",
    "\n",
    "Let's one-hot encode them along with the `Doors` column to make sure they're numerical, just as we did previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.0, 1.0, 0.0, ..., 1.0, 0.0, 71934.0],\n",
       "       [0.0, 0.0, 0.0, ..., 1.0, 0.0, 162665.0],\n",
       "       [0.0, 1.0, 0.0, ..., 1.0, 0.0, 42844.0],\n",
       "       ...,\n",
       "       [0.0, 0.0, 0.0, ..., 1.0, 0.0, 196225.0],\n",
       "       [0.0, 1.0, 0.0, ..., 1.0, 0.0, 133117.0],\n",
       "       [0.0, 1.0, 0.0, ..., 1.0, 0.0, 150582.0]], dtype=object)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now let's one hot encode the features with the same code as before \n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "categorical_features = [\"Make\", \"Colour\", \"Doors\"]\n",
    "\n",
    "one_hot = OneHotEncoder()\n",
    "\n",
    "transformer = ColumnTransformer([(\"one_hot\", \n",
    "                                  one_hot, \n",
    "                                  categorical_features)],\n",
    "                                remainder=\"passthrough\",\n",
    "                                sparse_threshold=0) # return a sparse matrix or not\n",
    "\n",
    "# Fill train and test values separately\n",
    "transformed_X_train = transformer.fit_transform(filled_X_train_df)\n",
    "transformed_X_test = transformer.transform(filled_X_test_df)\n",
    "\n",
    "# Check transformed and filled X_train\n",
    "transformed_X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Now our data is:\n",
    "1. All numerical\n",
    "2. No missing values\n",
    "\n",
    "Let's try and fit a model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.21229043336119102"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we've transformed X, let's see if we can fit a model\n",
    "np.random.seed(42)\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "model = RandomForestRegressor()\n",
    "\n",
    "# Make sure to use the transformed data (filled and one-hot encoded X data)\n",
    "model.fit(transformed_X_train, y_train)\n",
    "model.score(transformed_X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might have noticed this result is slightly different to before.\n",
    "\n",
    "Why do you think this is?\n",
    "\n",
    "It's because we've created our training and testing sets differently.\n",
    "\n",
    "We split the data into training and test sets *before* filling the missing values.\n",
    "\n",
    "Previously, we did the reverse, filled missing values *before* splitting the data into training and test sets.\n",
    "\n",
    "Doing this can lead to information from the training set leaking into the testing set.\n",
    "\n",
    "Remember, one of the most important concepts in machine learning is making sure your model doesn't see *any* testing data before evaluation.\n",
    "\n",
    "We'll keep practicing but for now, some of the main takeaways are:\n",
    "* Keep your training and test sets separate.\n",
    "* Most datasets you come across won't be in a form ready to immediately start using them with machine learning models. And some may take more preparation than others to get ready to use.\n",
    "* For most machine learning models, your data has to be numerical. This will involve converting whatever you're working with into numbers. This process is often referred to as **feature engineering** or **feature encoding**.\n",
    "* Some machine learning models aren't compatible with missing data. The process of filling missing data is referred to as **data imputation**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Choosing the right estimator/algorithm for your problem\n",
    "\n",
    "Once you've got your data ready, the next step is to choose an appropriate machine learning algorithm or model to find patterns in your data.\n",
    "\n",
    "Some things to note:\n",
    "* Scikit-Learn refers to machine learning models and algorithms as estimators.\n",
    "* **Classification problem** - predicting a category (heart disease or not).\n",
    "    * Sometimes you'll see `clf` (short for classifier) used as a classification estimator instance's variable name.\n",
    "* **Regression problem** - predicting a number (selling price of a car).\n",
    "* **Unsupervised problem (data with no labels)** - clustering (grouping unlabelled samples with other similar unlabelled samples).\n",
    "\n",
    "If you know what kind of problem you're working with, one of the next places you should look at is the [Scikit-Learn algorithm cheatsheet](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html).\n",
    "\n",
    "This cheatsheet gives you a bit of an insight into the algorithm you might want to use for the problem you're working on.\n",
    "\n",
    "It's important to remember, you don't have to explicitly know what each algorithm is doing on the inside to start using them. \n",
    "\n",
    "If you start to apply different algorithms but they don't seem to be working (not performing as well as you'd like), that's when you'd start to look deeper into each one.\n",
    "\n",
    "Let's check out the cheatsheet and follow it for some of the problems we're working on.\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map.png\" width=700/>\n",
    "\n",
    "You can see it's split into four main categories. Regression, classification, clustering and dimensionality reduction. Each has their own different purpose but the Scikit-Learn team has designed the library so the workflows for each are relatively similar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Picking a machine learning model for a regression problem\n",
    "\n",
    "Let's start with a regression problem (trying to predict a number). We'll use the [California Housing dataset](https://scikit-learn.org/stable/datasets/real_world.html#california-housing-dataset) built into Scikit-Learn's `datasets` module.\n",
    "\n",
    "The goal of the California Housing dataset is to predict a given district's median house value (in hundreds of thousands of dollars) on things like the age of the home, the number of rooms, the number of bedrooms, number of people living the home and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get California Housing dataset\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "housing = fetch_california_housing()\n",
    "housing; # gets downloaded as dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since it's in a dictionary, let's turn it into a DataFrame so we can inspect it better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.3252</td>\n",
       "      <td>41.0</td>\n",
       "      <td>6.984127</td>\n",
       "      <td>1.023810</td>\n",
       "      <td>322.0</td>\n",
       "      <td>2.555556</td>\n",
       "      <td>37.88</td>\n",
       "      <td>-122.23</td>\n",
       "      <td>4.526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.3014</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6.238137</td>\n",
       "      <td>0.971880</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>2.109842</td>\n",
       "      <td>37.86</td>\n",
       "      <td>-122.22</td>\n",
       "      <td>3.585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.2574</td>\n",
       "      <td>52.0</td>\n",
       "      <td>8.288136</td>\n",
       "      <td>1.073446</td>\n",
       "      <td>496.0</td>\n",
       "      <td>2.802260</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.24</td>\n",
       "      <td>3.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.6431</td>\n",
       "      <td>52.0</td>\n",
       "      <td>5.817352</td>\n",
       "      <td>1.073059</td>\n",
       "      <td>558.0</td>\n",
       "      <td>2.547945</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.8462</td>\n",
       "      <td>52.0</td>\n",
       "      <td>6.281853</td>\n",
       "      <td>1.081081</td>\n",
       "      <td>565.0</td>\n",
       "      <td>2.181467</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.422</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "0  8.3252      41.0  6.984127   1.023810       322.0  2.555556     37.88   \n",
       "1  8.3014      21.0  6.238137   0.971880      2401.0  2.109842     37.86   \n",
       "2  7.2574      52.0  8.288136   1.073446       496.0  2.802260     37.85   \n",
       "3  5.6431      52.0  5.817352   1.073059       558.0  2.547945     37.85   \n",
       "4  3.8462      52.0  6.281853   1.081081       565.0  2.181467     37.85   \n",
       "\n",
       "   Longitude  target  \n",
       "0    -122.23   4.526  \n",
       "1    -122.22   3.585  \n",
       "2    -122.24   3.521  \n",
       "3    -122.25   3.413  \n",
       "4    -122.25   3.422  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = pd.DataFrame(housing[\"data\"], columns=housing[\"feature_names\"])\n",
    "housing_df[\"target\"] = pd.Series(housing[\"target\"])\n",
    "housing_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20640"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How many samples?\n",
    "len(housing_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful, our goal here is to use the feature columns, such as:\n",
    "* `MedInc` - median income in block group\n",
    "* `HouseAge` - median house age in block group\n",
    "* `AveRooms` - average number of rooms per household\n",
    "* `AveBedrms` - average number of bedrooms per household\n",
    "\n",
    "To predict the `target` column which expresses the median house value for specfici California districts in hundreds of thousands of dollars (e.g. 4.526 = $452,600). \n",
    "\n",
    "In essence, each row is a different district in California (the data) and we're trying to build a model to predict the median house value in that distract (the target/label) given a series of attributes about the houses in that district.\n",
    "\n",
    "Since we have data and labels, this is a supervised learning problem. \n",
    "\n",
    "And since we're trying to predict a number, it's a regression problem.\n",
    "\n",
    "Knowing these two things, how do they line up on the Scikit-Learn machine learning algorithm cheat-sheet?\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map-cheatsheet-california-housing-ridge.png\" width=700/>\n",
    "\n",
    "Following the map through, knowing what we know, it suggests we try [`RidgeRegression`](https://scikit-learn.org/stable/modules/linear_model.html#ridge-regression). Let's chek it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5758549611440125"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the Ridge model class from the linear_model module\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into features (X) and labels (y)\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Institate and fit the model (on the training set)\n",
    "model = Ridge()\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Check the score of the model (on the test set)\n",
    "# The default score() metirc of regression aglorithms is R^2\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if `RidgeRegression` didn't work? Or what if we wanted to improve our results?\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map-cheatsheet-california-housing-ensemble.png\" width=700/>\n",
    "\n",
    "Following the diagram, the next step would be to try [`EnsembleRegressors`](https://scikit-learn.org/stable/modules/ensemble.html). \n",
    "\n",
    "Ensemble is another word for multiple models put together to make a decision.\n",
    "\n",
    "One of the most common and useful ensemble methods is the [Random Forest](https://scikit-learn.org/stable/modules/ensemble.html#forest). Known for its fast training and prediction times and adaptibility to different problems.\n",
    "\n",
    "The basic premise of the Random Forest is to combine a number of different decision trees, each one random from the other and make a prediction on a sample by averaging the result of each decision tree.\n",
    "\n",
    "An in-depth discussion of the Random Forest algorithm is beyond the scope of this notebook but if you're interested in learning more, [An Implementation and Explanation of the Random Forest in Python](https://towardsdatascience.com/an-implementation-and-explanation-of-the-random-forest-in-python-77bf308a9b76) by Will Koehrsen is a great read.\n",
    "\n",
    "Since we're working with regression, we'll use Scikit-Learn's [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html).\n",
    "\n",
    "We can use the exact same workflow as above. Except for changing the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8059809073051385"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the RandomForestRegressor model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into features (X) and labels (y)\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Institate and fit the model (on the training set)\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Check the score of the model (on the test set)\n",
    "# The default score metirc of regression aglorithms is R^2\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! \n",
    "\n",
    "We get a good boost in score on the test set by changing the model.\n",
    "\n",
    "This is another incredibly important concept in machine learning, if at first something doesn't achieve what you'd like, *experiment, experiment, experiment!*\n",
    "\n",
    "At first, the Scikit-Learn algorithm diagram can seem confusing. \n",
    "\n",
    "But once you get a little practice applying different models to different problems, you'll start to pick up which sorts of algorithms do better with different types of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Picking a machine learning model for a classification problem\n",
    "\n",
    "Now, let's check out the choosing process for a classification problem.\n",
    "\n",
    "Say you were trying to predict whether or not a patient had heart disease based on their medical records.\n",
    "\n",
    "The dataset in `../data/heart-disease.csv` (or at [`heart-disease.csv`](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/heart-disease.csv)) contains data for just that problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   0     1       1  \n",
       "1   0     2       1  \n",
       "2   0     2       1  \n",
       "3   0     2       1  \n",
       "4   0     2       1  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# heart_disease = pd.read_csv(\"../data/heart-disease.csv\") # load from local directory\n",
    "heart_disease = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\") # load directly from URL (source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/heart-disease.csv)\n",
    "heart_disease.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "303"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How many samples are there?\n",
    "len(heart_disease)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the California Housing dataset, here we want to use all of the available data to predict the target column (1 for if a patient has heart disease and 0 for if they don't).\n",
    "\n",
    "So what do we know?\n",
    "\n",
    "We've got 303 samples (1 row = 1 sample) and we're trying to predict whether or not a patient has heart disease.\n",
    "\n",
    "Because we're trying to predict whether each sample is one thing or another, we've got a classification problem.\n",
    "\n",
    "Let's see how it lines up with our [Scikit-Learn algorithm cheat-sheet](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html).\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map-cheatsheet-heart-disease-linear-svc.png\" width=700/>\n",
    "\n",
    "Following the cheat-sheet we end up at [`LinearSVC`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) which stands for Linear Support Vector Classifier. Let's try it on our data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8688524590163934"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import LinearSVC from the svm module\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into X (features/data) and y (target/labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Instantiate and fit the model (on the training set)\n",
    "clf = LinearSVC(max_iter=1000, # iterations on the data, 1000 is the default\n",
    "                dual=\"auto\") # dual=\"auto\" chooses best parameters for the model automatically\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Check the score of the model (on the test set)\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Straight out of the box (with no tuning or improvements) our model achieves over 85% accruacy!\n",
    "\n",
    "Although this is a sensational result to begin with, let's check out the diagram and see what other models we might use.\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map-cheatsheet-heart-disease-ensemble.png\" width=700/>\n",
    "\n",
    "Following the path (and skipping a few, don't worry, we'll get to this) we come up to [`EnsembleMethods`](https://scikit-learn.org/stable/modules/ensemble.html) again. \n",
    "\n",
    "Except this time, we'll be looking at ensemble classifiers instead of regressors.\n",
    "\n",
    "Remember our [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) from above? \n",
    "\n",
    "We'll it has a dance partner, [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) which is an ensemble based machine model learning model for classification. \n",
    "\n",
    "You might be able to guess what we can use it for (hint: classification problems).\n",
    "\n",
    "Let's try!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the RandomForestClassifier model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into X (features/data) and y (target/labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Instantiate and fit the model (on the training set)\n",
    "clf = RandomForestClassifier(n_estimators=100) # 100 is the default, but you could try 1000 and see what happens\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Check the score of the model (on the test set)\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm, it looks like the default hyperparameters of `RandomForestClassifier` don't perform as well as `LinearSVC`.\n",
    "\n",
    "Other than trying another classification model, we could start to run experiments to try and improve these models via [hyperparameter tuning](http://scikit-learn.org/stable/modules/grid_search.html). \n",
    "\n",
    "Hyperparameter tuning is fancy term for adjusting some settings on a model to try and make it better. \n",
    "\n",
    "It usually happens once you've found a decent baseline model that you'd like to improve upon.\n",
    "\n",
    "In this case, we could take either the `RandomForestClassifier` or the `LinearSVC` and try and improve it with hyperparameter tuning (which we'll see later on).\n",
    "\n",
    "For example, you could try and take the `n_estimators` parameter (the number of trees in the forest) of `RandomForestClassifier` and change it from `100` (default) to `1000` and see what happens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about the other models?\n",
    "\n",
    "Looking at the Scikit-Learn aglorithm cheat-sheet and the examples above, you may have noticed we've skipped a few models.\n",
    "\n",
    "Why?\n",
    "\n",
    "The first reason is time. \n",
    "\n",
    "Covering every single one would take a fair bit longer than what we've done here. And the second one is the effectiveness of ensemble methods.\n",
    "\n",
    "A little tidbit for modelling in machine learning is:\n",
    "* If you have structured data (tables, spreadsheets or dataframes), use ensemble methods, such as, a Random Forest.\n",
    "* If you have unstructured data (text, images, audio, things not in tables), use deep learning or transfer learning (see the ZTM [TensorFlow](https://dbourke.link/ZTMTFcourse?ref=mrdbourke.com) and [PyTorch](https://dbourke.link/ZTMPyTorch?ref=mrdbourke.com) courses for more on deep learning).\n",
    "\n",
    "For this notebook, we're focused on structured data, which is why the Random Forest has been our model of choice.\n",
    "\n",
    "If you'd like to learn more about the Random Forest and why it's the war horse of machine learning, check out these resources:\n",
    "* [Random Forest Wikipedia](https://en.wikipedia.org/wiki/Random_forest)\n",
    "* [An Implementation and Explanation of the Random Forest in Python](https://towardsdatascience.com/an-implementation-and-explanation-of-the-random-forest-in-python-77bf308a9b76) by Will Koehrsen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment until something works\n",
    "\n",
    "The beautiful thing is, the way the Scikit-Learn API is designed, once you know the way with one model, using another is much the same.\n",
    "\n",
    "And since a big part of being a machine learning engineer or data scientist is experimenting, you might want to try out some of the other models on the cheat-sheet and see how you go. The more you can reduce the time between experiments, the better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Fit the model to data and using it to make predictions\n",
    "\n",
    "Now you've chosen a model, the next step is to have it learn from the data so it can be used for predictions in the future.\n",
    "\n",
    "If you've followed through, you've seen a few examples of this already."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Fitting a model to data\n",
    "\n",
    "In Scikit-Learn, the process of having a machine learning model learn patterns from a dataset involves calling the `fit()` method and passing it data, such as, `fit(X, y)`.\n",
    "\n",
    "Where `X` is a feature array and `y` is a target array.\n",
    "\n",
    "Other names for `X` include:\n",
    "* Data\n",
    "* Feature variables\n",
    "* Features\n",
    "\n",
    "Other names for `y` include:\n",
    "* Labels\n",
    "* Target variable\n",
    "\n",
    "For supervised learning there is usually an `X` and `y`. \n",
    "\n",
    "For unsupervised learning, there's no `y` (no labels).\n",
    "\n",
    "Let's revisit the example of using patient data (`X`) to predict whether or not they have heart disease (`y`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the RandomForestClassifier model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into X (features/data) and y (target/labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Instantiate the model (on the training set)\n",
    "clf = RandomForestClassifier(n_estimators=100)\n",
    "\n",
    "# Call the fit method on the model and pass it training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Check the score of the model (on the test set)\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening here?\n",
    "\n",
    "Calling the `fit()` method will cause the machine learning algorithm to attempt to find patterns between `X` and `y`. Or if there's no `y`, it'll only find the patterns within `X`.\n",
    "\n",
    "Let's see `X`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  \n",
       "0   0     1  \n",
       "1   0     2  \n",
       "2   0     2  \n",
       "3   0     2  \n",
       "4   0     2  "
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    1\n",
       "Name: target, dtype: int64"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Passing `X` and `y` to `fit()` will cause the model to go through all of the examples in `X` (data) and see what their corresponding `y` (label) is.\n",
    "\n",
    "How the model does this is different depending on the model you use.\n",
    "\n",
    "Explaining the details of each would take an entire textbook. \n",
    "\n",
    "For now, you could imagine it similar to how you would figure out patterns if you had enough time. \n",
    "\n",
    "You'd look at the feature variables, `X`, the `age`, `sex`, `chol` (cholesterol) and see what different values led to the labels, `y`, `1` for heart disease, `0` for not heart disease.\n",
    "\n",
    "This concept, regardless of the problem, is similar throughout all of machine learning.\n",
    "\n",
    "**During training (finding patterns in data):**\n",
    "\n",
    "A machine learning algorithm looks at a dataset, finds patterns, tries to use those patterns to predict something and corrects itself as best it can with the available data and labels. It stores these patterns for later use.\n",
    "\n",
    "**During testing or in production (using learned patterns):**\n",
    "\n",
    "A machine learning algorithm uses the patterns its previously learned in a dataset to make a prediction on some unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Making predictions using a machine learning model\n",
    "Now we've got a trained model, one which has hoepfully learned patterns in the data, you'll want to use it to make predictions.\n",
    "\n",
    "Scikit-Learn enables this in several ways. \n",
    "\n",
    "Two of the most common and useful are [`predict()`](https://github.com/scikit-learn/scikit-learn/blob/5f3c3f037/sklearn/multiclass.py#L299) and [`predict_proba()`](https://github.com/scikit-learn/scikit-learn/blob/5f3c3f037/sklearn/linear_model/_logistic.py#L1617).\n",
    "\n",
    "Let's see them in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0,\n",
       "       1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use a trained model to make predictions\n",
    "clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given data in the form of `X`, the `predict()` function returns labels in the form of `y`.\n",
    "\n",
    "> **Note:** For the `predict()` function to work, it must be passed `X` (data) in the same format the model was trained on. For example, if a model was trained on 10 features formatted in a certain way, predictions should be made on data with 10 features fortmatted in a certain way. Anything different and it will return an error. \n",
    "\n",
    "It's standard practice to save these predictions to a variable named something like `y_preds` for later comparison to `y_test` or `y_true` (usually same as `y_test` just another name)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.8524590163934426)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compare predictions to truth\n",
    "y_preds = clf.predict(X_test)\n",
    "np.mean(y_preds == y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way evaluating predictions (comparing them to the truth labels) is with Scikit-Learn's [`sklearn.metrics` module](http://scikit-learn.org/stable/modules/model_evaluation.html).\n",
    "\n",
    "Inside, you'll find method such as [`accuracy_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html), which is the default evaluation metric for classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`predict_proba()` returns the probabilities (proba is short for probability) of a classification label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.89, 0.11],\n",
       "       [0.49, 0.51],\n",
       "       [0.43, 0.57],\n",
       "       [0.84, 0.16],\n",
       "       [0.18, 0.82]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return probabilities rather than labels\n",
    "clf.predict_proba(X_test[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 0, 1])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return labels\n",
    "clf.predict(X_test[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`predict_proba()` returns an array of five arrays each containing two values.\n",
    "\n",
    "Each number is the probability of a label given a sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.89, 0.11]])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find prediction probabilities for 1 sample\n",
    "clf.predict_proba(X_test[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This output means for the sample `X_test[:1]`, the model is predicting label 0 (index 0) with a probability score of 0.9.\n",
    "\n",
    "Because the highest probability score is at index `0` (and it's over 0.5), when using `predict()`, a label of `0` is assigned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return the label for 1 sample\n",
    "clf.predict(X_test[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where does 0.5 come from?\n",
    "\n",
    "Because our problem is a binary classification task (heart disease or not heart disease), predicting a label with 0.5 probability every time would be the same as a coin toss (guessing 50/50 every time). \n",
    "\n",
    "Therefore, once the prediction probability of a sample passes 0.5 for a certain label, it's assigned that label.\n",
    "\n",
    "`predict()` can also be used for regression models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the RandomForestRegressor model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into features (X) and labels (y)\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Institate and fit the model (on the training set)\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_preds = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can evaluate our regression model by using [`sklearn.metrics.mean_absolute_error`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) which returns the average error across all samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.3270458119670544)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compare the predictions to the truth\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "mean_absolute_error(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've seen how to get a model how to find patterns in data using the `fit()` function and make predictions using what its learned using the `predict()` and `predict_proba()` functions, it's time to evaluate those predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluating a model\n",
    "\n",
    "Once you've trained a model, you'll want a way to measure how trustworthy its predictions are.\n",
    "\n",
    "Across the board, the main idea of evaluating a model is to **compare the model's predictions to what they should've ideally been** (the truth labels).\n",
    "\n",
    "Scikit-Learn implements 3 different methods of evaluating models.\n",
    "\n",
    "1. The `score()` method. Calling `score()` on a model instance will return a metric assosciated with the type of model you're using. The metric depends on which model you're using.\n",
    "2. The `scoring` parameter. This parameter can be passed to methods such as [`cross_val_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score) or [`GridSearchCV()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to tell Scikit-Learn to use a specific type of scoring metric.\n",
    "3. Problem-specific metric functions available in [`sklearn.metrics`](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics). Similar to how the `scoring` parameter can be passed different scoring functions, Scikit-Learn implements these as stand alone functions.\n",
    "\n",
    "The scoring function you use will also depend on the problem you're working on.\n",
    "\n",
    "Classification problems have different evaluation metrics and scoring functions to regression problems.\n",
    "\n",
    "Let's look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 General model evaluation with `score()`\n",
    "\n",
    "If we bring down the code from our previous classification problem (building a classifier to predict whether or not someone has heart disease based on their medical records).\n",
    "\n",
    "We can see the `score()` method come into play."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the RandomForestClassifier model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into X (features/data) and y (target/labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Instantiate the model (on the training set)\n",
    "clf = RandomForestClassifier(n_estimators=100)\n",
    "\n",
    "# Call the fit method on the model and pass it training data\n",
    "clf.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model has been fit on the training data (`X_train`, `y_train`), we can call the `score()` method on it and evaluate our model on the test data, data the model has never seen before (`X_test`, `y_test`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the score of the model (on the test set)\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each model in Scikit-Learn implements a default metric for `score()` which is suitable for the problem.\n",
    "\n",
    "For example:\n",
    "* Classifier models generally use [`metrics.accuracy_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) as the default `score()` metric.\n",
    "* Regression models generally use [`metrics.r2_score`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html) as the default `score()` metric.\n",
    "* There many more [classification](https://scikit-learn.org/stable/modules/classes.html#classification-metrics) and [regression](https://scikit-learn.org/stable/modules/classes.html#regression-metrics) specific metrics implemented in `sklearn.metrics`.\n",
    "\n",
    "Because `clf` is an instance of `RandomForestClassifier`, the `score()` method uses mean accuracy as its score method.\n",
    "\n",
    "You can find this by pressing **SHIFT + TAB** (inside a Jupyter Notebook, may be different elsewhere) within the brackets of `score()` when called on a model instance.\n",
    "\n",
    "Behind the scenes, `score()` makes predictions on `X_test` using the trained model and then compares those predictions to the actual labels `y_test`.\n",
    "\n",
    "A classification model which predicts everything 100% correct would receive an accuracy score of 1.0 (or 100%).\n",
    "\n",
    "Our model doesn't get everything correct, but at ~85% accuracy (0.85 * 100), it's still far better than guessing.\n",
    "\n",
    "Let's do the same but with the regression code from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the RandomForestRegressor model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into features (X) and labels (y)\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Institate and fit the model (on the training set)\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to the consistent design of the Scikit-Learn library, we can call the same `score()` method on `model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8059809073051385"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the score of the model (on the test set)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `model` is an instance of [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html). \n",
    "\n",
    "And since it's a regression model, the default metric built into `score()` is the coefficient of determination or R^2 (pronounced R-sqaured).\n",
    "\n",
    "Remember, you can find this by pressing **SHIFT + TAB** within the brackets of `score()` when called on a model instance.\n",
    "\n",
    "The best possible value here is 1.0, this means the model predicts the target regression values exactly.\n",
    "\n",
    "Calling the `score()` method on any model instance and passing it test data is a good quick way to see how your model is going.\n",
    "\n",
    "However, when you get further into a problem, it's likely you'll want to start using more powerful metrics to evaluate your models performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Evaluating your models using the `scoring` parameter \n",
    "\n",
    "The next step up from using `score()` is to use a custom `scoring` parameter with [`cross_val_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score) or [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html).\n",
    "\n",
    "As you may have guessed, the `scoring` parameter you set will be different depending on the problem you're working on.\n",
    "\n",
    "We'll see some specific examples of different parameters in a moment but first let's check out `cross_val_score()`.\n",
    "\n",
    "To do so, we'll copy the heart disease classification code from above and then add another line at the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import cross_val_score from the model_selection module\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Import the RandomForestClassifier model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split the data into X (features/data) and y (target/labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Instantiate the model (on the training set)\n",
    "clf = RandomForestClassifier(n_estimators=100)\n",
    "\n",
    "# Call the fit method on the model and pass it training data\n",
    "clf.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `cross_val_score()` is slightly different to `score()`. \n",
    "\n",
    "Let's see a code example first and then we'll go through the details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using score()\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.81967213, 0.86885246, 0.81967213, 0.78333333, 0.76666667])"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using cross_val_score()\n",
    "cross_val_score(clf, X, y, cv=5) # cv = number of splits to test (5 by default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening here?\n",
    "\n",
    "The first difference you might notice is `cross_val_score()` returns an array where as `score()` only returns a single number.\n",
    "\n",
    "`cross_val_score()` returns an array because of a parameter called `cv`, which stands for cross-validation.\n",
    "\n",
    "When `cv` isn't set, `cross_val_score()` will return an array of 5 numbers by default (`cv=None` is the same as setting `cv=5`).\n",
    "\n",
    "Remember, you can see the parameters of a function using **SHIFT + TAB** (inside a Jupyter Notebook) from within the brackets.\n",
    "\n",
    "But wait, you might be thinking, what even is cross-validation?\n",
    "\n",
    "A visual might be able to help.\n",
    "\n",
    "<img src='../images/sklearn-cross-validation.png' width=600/>\n",
    "\n",
    "We've dealt with Figure 1.0 before using `score(X_test, y_test)`. \n",
    "\n",
    "But looking deeper into this, if a model is trained using the training data or 80% of samples, this means 20% of samples aren't used for the model to learn anything.\n",
    "\n",
    "This also means depending on what 80% is used to train on and what 20% is used to evaluate the model, it may achieve a score which doesn't reflect the entire dataset. \n",
    "\n",
    "For example, if a lot of easy examples are in the 80% training data, when it comes to test on the 20%, your model may perform poorly. \n",
    "\n",
    "The same goes for the reverse.\n",
    "\n",
    "Figure 2.0 shows 5-fold cross-validation, a method which tries to provide a solution to:\n",
    "\n",
    "1. Not training on all the data (always keeping training and test sets separate).\n",
    "2. Avoiding getting lucky scores on single splits of the data.\n",
    "\n",
    "Instead of training only on 1 training split and evaluating on 1 testing split, 5-fold cross-validation does it 5 times. \n",
    "\n",
    "On a different split each time, returning a score for each.\n",
    "\n",
    "Why 5-fold?\n",
    "\n",
    "The actual name of this setup K-fold cross-validation. Where K is an abitrary number. We've used 5 because it looks nice visually, and it is the default value in [`sklearn.model_selection.cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html).\n",
    "\n",
    "Figure 2.0 is what happens when we run the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.83606557, 0.8852459 , 0.7704918 , 0.8       , 0.8       ])"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 5-fold cross-validation\n",
    "cross_val_score(clf, X, y, cv=5) # cv is equivalent to K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we set `cv=5` (5-fold cross-validation), we get back 5 different scores instead of 1.\n",
    "\n",
    "Taking the mean of this array gives us a more in-depth idea of how our model is performing by converting the 5 scores into one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8524590163934426, np.float64(0.8248087431693989))"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "# Single training and test split score\n",
    "clf_single_score = clf.score(X_test, y_test)\n",
    "\n",
    "# Take mean of 5-fold cross-validation\n",
    "clf_cross_val_score = np.mean(cross_val_score(clf, X, y, cv=5))\n",
    "\n",
    "clf_single_score, clf_cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, the average `cross_val_score()` is slightly lower than single value returned by `score()`.\n",
    "\n",
    "In this case, if you were asked to report the accuracy of your model, even though it's lower, you'd prefer the cross-validated metric over the non-cross-validated metric.\n",
    "\n",
    "Wait?\n",
    "\n",
    "We haven't used the `scoring` parameter at all.\n",
    "\n",
    "By default, it's set to `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.78688525, 0.86885246, 0.80327869, 0.78333333, 0.76666667])"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(clf, X, y, cv=5, scoring=None) # default scoring value, this can be set to other scoring metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** If you notice different scores each time you call `cross_val_score`, this is because each data split is random every time. So the model may achieve higher/lower scores on different splits of the data. To get reproducible scores, you can set the random seed.\n",
    "\n",
    "When `scoring` is set to `None` (by default), it uses the same metric as `score()` for whatever model is passed to `cross_val_score()`.\n",
    "\n",
    "In this case, our model is `clf` which is an instance of `RandomForestClassifier` which uses mean accuracy as the default `score()` metric.\n",
    "\n",
    "You can change the evaluation score `cross_val_score()` uses by changing the `scoring` parameter.\n",
    "\n",
    "And as you might have guessed, different problems call for different evaluation scores.\n",
    "\n",
    "The [Scikit-Learn documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter) outlines a vast range of evaluation metrics for different problems but let's have a look at a few."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.1 Classification model evaluation metrics\n",
    "\n",
    "Four of the main evaluation metrics/methods you'll come across for classification models are:\n",
    "\n",
    "1. [Accuracy](https://developers.google.com/machine-learning/crash-course/classification/accuracy)\n",
    "2. [Area under ROC curve](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc) (receiver operating characteristic curve)\n",
    "3. [Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix)\n",
    "4. [Classification report](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)\n",
    "\n",
    "Let's have a look at each of these. We'll bring down the classification code from above to go through some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8524590163934426"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import cross_val_score from the model_selection module\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "clf = RandomForestClassifier()\n",
    "clf.fit(X_train, y_train)\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy\n",
    "Accuracy is the default metric for the `score()` function within each of Scikit-Learn's classifier models. And it's probably the metric you'll see most often used for classification problems.\n",
    "\n",
    "However, we'll see in a second how it may not always be the best metric to use.\n",
    "\n",
    "Scikit-Learn returns accuracy as a decimal but you can easily convert it to a percentage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Heart Disease Classifier Accuracy: 85.25%\n"
     ]
    }
   ],
   "source": [
    "# Accuracy as percentage\n",
    "print(f\"Heart Disease Classifier Accuracy: {clf.score(X_test, y_test) * 100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Area Under Receiver Operating Characteristic (ROC) Curve\n",
    "If this one sounds like a mouthful, its because reading the full name is.\n",
    "\n",
    "It's usually referred to as AUC for Area Under Curve and the curve they're talking about is the Receiver Operating Characteristic or ROC for short.\n",
    "\n",
    "So if hear someone talking about AUC or ROC, they're probably talking about what follows.\n",
    "\n",
    "ROC curves are a comparison of true postive rate (tpr) versus false positive rate (fpr).\n",
    "\n",
    "For clarity:\n",
    "* True positive = model predicts 1 when truth is 1\n",
    "* False positive = model predicts 1 when truth is 0\n",
    "* True negative = model predicts 0 when truth is 0\n",
    "* False negative = model predicts 0 when truth is 1\n",
    "\n",
    "Now we know this, let's see one. Scikit-Learn lets you calculate the information required for a ROC curve using the [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.03448276, 0.03448276, 0.03448276, 0.03448276, 0.06896552,\n",
       "       0.06896552, 0.10344828, 0.13793103, 0.13793103, 0.17241379,\n",
       "       0.17241379, 0.27586207, 0.4137931 , 0.48275862, 0.55172414,\n",
       "       0.65517241, 0.72413793, 0.72413793, 0.82758621, 1.        ])"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# Make predictions with probabilities\n",
    "y_probs = clf.predict_proba(X_test)\n",
    "\n",
    "# Keep the probabilites of the positive class only\n",
    "y_probs = y_probs[:, 1]\n",
    "\n",
    "# Calculate fpr, tpr and thresholds\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_probs)\n",
    "\n",
    "# Check the false positive rate\n",
    "fpr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these on their own doesn't make much sense. It's much easier to see their value visually. \n",
    "\n",
    "Let's create a helper function to make a ROC curve given the false positive rates (`fpr`) and true positive rates (`tpr`).\n",
    "\n",
    "> **Note:** As of Scikit-Learn 1.2+, there is functionality of plotting a ROC curve. You can find this under [`sklearn.metrics.RocCurveDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.html#sklearn-metrics-roccurvedisplay)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB2vElEQVR4nO3dd1hT1/8H8HcCJOyhKKKiiHsjqNS9UFyorQNHFdFq66rVuhfaqjjqat0Tt+Bqnbi1rqoVsS4cuBVQRKbM5Pz+8Ee+IqAEgct4v54nj+bk3Jt3LoF8cu6598qEEAJEREREBYRc6gBERERE2YnFDRERERUoLG6IiIioQGFxQ0RERAUKixsiIiIqUFjcEBERUYHC4oaIiIgKFBY3REREVKCwuCEiIqIChcUN5TpbW1v0799f6hiFTvPmzdG8eXOpY3zW9OnTIZPJEBYWJnWUPEcmk2H69OnZsq7Hjx9DJpPB29s7W9YHAJcvX4ZCocCTJ0+ybZ3ZrWfPnujRo4fUMSiHsbgpYLy9vSGTyTQ3XV1dlCpVCv3798eLFy+kjpenxcbG4tdff0WtWrVgaGgIMzMzNGnSBJs2bUJ+uUrJ7du3MX36dDx+/FjqKGmoVCps2LABzZs3R5EiRaBUKmFrawsPDw/8+++/UsfLFtu2bcPixYuljpFKbmaaPHkyevXqhbJly2ramjdvnupvkoGBAWrVqoXFixdDrVanu543b95g7NixqFy5MvT19VGkSBG4uLjgwIEDGT53VFQUZsyYgdq1a8PY2BgGBgaoUaMGxo8fj5cvX2r6jR8/Hrt378b169cz/boKw3u3wBFUoGzYsEEAEL/88ovYvHmzWLNmjRg4cKDQ0dER5cuXF3FxcVJHFPHx8SIxMVHqGKmEhISI6tWrC7lcLnr37i1WrVollixZIpo2bSoACDc3N5GcnCx1zM/auXOnACBOnTqV5rGEhASRkJCQ+6GEEO/evRNt27YVAETTpk3F/Pnzxbp168TUqVNF5cqVhUwmE8+ePRNCCOHp6SkAiNevX0uS9Ut06NBBlC1bNsfWHxcXJ5KSkrRaJqNMarVaxMXFZdv7+tq1awKAuHDhQqr2Zs2aidKlS4vNmzeLzZs3i0WLFol69eoJAGLSpElp1hMYGChKlSolFAqF+P7778WaNWvE/Pnzhb29vQAgxowZk2aZoKAgUa5cOaGjoyN69uwpli5dKlavXi2GDx8uihYtKipWrJiqf/369UXfvn0z9bq0ee9S3sHipoBJKW6uXLmSqn38+PECgPDx8ZEombTi4uKESqXK8HEXFxchl8vFX3/9leaxMWPGCABizpw5ORkxXTExMVr1/1RxI6Vhw4YJAGLRokVpHktOThbz58/P1eJGrVaLd+/eZft6c6K4UalUX/SlJKcLrhQ//vijKFOmjFCr1anamzVrJqpXr56qLS4uTpQtW1aYmJikKq4SExNFjRo1hKGhofjnn39SLZOcnCzc3NwEALFjxw5Ne1JSkqhdu7YwNDQUZ8+eTZMrMjIyTRH122+/CSMjIxEdHf3Z16XNe/dLfOnPmVJjcVPAZFTcHDhwQAAQs2fPTtV+584d0bVrV2FhYSGUSqVwdHRM9wP+7du34qeffhJly5YVCoVClCpVSvTt2zfVB1B8fLyYNm2aKF++vFAoFKJ06dJi7NixIj4+PtW6ypYtK9zd3YUQQly5ckUAEN7e3mme08/PTwAQ+/fv17Q9f/5ceHh4iOLFiwuFQiGqVasm1q1bl2q5U6dOCQBi+/btYvLkyaJkyZJCJpOJt2/fprvNLl68KACIAQMGpPt4UlKSqFixorCwsNB8ID569EgAEPPnzxcLFy4UZcqUEfr6+qJp06bixo0badaRme2c8rM7ffq0GDJkiChWrJgwNzcXQgjx+PFjMWTIEFGpUiWhr68vihQpIrp16yYePXqUZvmPbymFTrNmzUSzZs3SbCcfHx8xc+ZMUapUKaFUKkXLli3F/fv307yGpUuXinLlygl9fX1Rr1498ffff6dZZ3qePXsmdHV1RevWrT/ZL0VKcXP//n3h7u4uzMzMhKmpqejfv7+IjY1N1Xf9+vWiRYsWolixYkKhUIiqVauK5cuXp1ln2bJlRYcOHYSfn59wdHQUSqVS82GV2XUIIcShQ4dE06ZNhbGxsTAxMRF169YVW7duFUK8374fb/sPi4rM/n4AEMOGDRNbtmwR1apVE7q6umLv3r2axzw9PTV9o6KixMiRIzW/l8WKFRPOzs7i6tWrn82U8h7esGFDque/c+eO6N69u7C0tBT6+vqiUqVK6Y6wfKxMmTKif//+adrTK26EEKJbt24CgHj58qWmbfv27ZqR5/REREQIc3NzUaVKFU3bjh07BAAxa9asz2ZMcf36dQFA7Nmz55P9tH3vuru7p1tIprynP5Tez9nX11dYWFikux0jIyOFUqkUP//8s6Yts++pwkg32/dzUZ6UMgfDwsJC03br1i00atQIpUqVwoQJE2BkZARfX1906dIFu3fvxtdffw0AiImJQZMmTXDnzh0MGDAADg4OCAsLw759+/D8+XNYWlpCrVajU6dOOHfuHAYPHoyqVavixo0bWLRoEe7du4c///wz3Vx169aFnZ0dfH194e7unuoxHx8fWFhYwMXFBQAQGhqKr776CjKZDMOHD0exYsVw+PBhDBw4EFFRUfjpp59SLf/rr79CoVBgzJgxSEhIgEKhSDfD/v37AQD9+vVL93FdXV307t0bM2bMwPnz5+Hs7Kx5bNOmTYiOjsawYcMQHx+PJUuWoGXLlrhx4wasrKy02s4phg4dimLFimHatGmIjY0FAFy5cgUXLlxAz549Ubp0aTx+/BgrVqxA8+bNcfv2bRgaGqJp06b48ccf8fvvv2PSpEmoWrUqAGj+zcicOXMgl8sxZswYREZGYt68eejTpw8uXbqk6bNixQoMHz4cTZo0wahRo/D48WN06dIFFhYWKF269CfXf/jwYSQnJ6Nv376f7PexHj16oFy5cvDy8oK/vz/Wrl2L4sWLY+7cualyVa9eHZ06dYKuri7279+PoUOHQq1WY9iwYanWd/fuXfTq1Qvff/89Bg0ahMqVK2u1Dm9vbwwYMADVq1fHxIkTYW5ujmvXrsHPzw+9e/fG5MmTERkZiefPn2PRokUAAGNjYwDQ+vfj5MmT8PX1xfDhw2FpaQlbW9t0t9EPP/yAXbt2Yfjw4ahWrRrevHmDc+fO4c6dO3BwcPhkpvT8999/aNKkCfT09DB48GDY2toiKCgI+/fvx6xZszJc7sWLF3j69CkcHBwy7POxlAnN5ubmmrbP/S6amZmhc+fO2LhxIx48eIAKFSpg3759AKDV+6tatWowMDDA+fPn0/z+fSir793M+vjnXLFiRXz99dfYs2cPVq1alepv1p9//omEhAT07NkTgPbvqUJH6uqKslfKt/fjx4+L169fi2fPnoldu3aJYsWKCaVSmWr4tFWrVqJmzZqpqny1Wi0aNmyYah/1tGnTMvyWkzIEvXnzZiGXy9MMC69cuVIAEOfPn9e0fThyI4QQEydOFHp6eiI8PFzTlpCQIMzNzVONpgwcOFBYW1uLsLCwVM/Rs2dPYWZmphlVSRmRsLOzy9Suhy5duggAGY7sCCHEnj17BADx+++/CyH+963XwMBAPH/+XNPv0qVLAoAYNWqUpi2z2znlZ9e4ceM08yDSex0pI06bNm3StH1qt1RGIzdVq1ZNNRdnyZIlAoBmBCohIUEULVpU1KtXL9V8D29vbwHgsyM3o0aNEgDEtWvXPtkvRcq33I9H0r7++mtRtGjRVG3pbRcXFxdhZ2eXqq1s2bICgPDz80vTPzPriIiIECYmJsLJySnNroMPd8NktAtIm98PAEIul4tbt26lWQ8+GrkxMzMTw4YNS9PvQxllSm/kpmnTpsLExEQ8efIkw9eYnuPHj6cZZU3RrFkzUaVKFfH69Wvx+vVrERgYKMaOHSsAiA4dOqTqa29vL8zMzD75XAsXLhQAxL59+4QQQtSpU+ezy6SnUqVKol27dp/so+17V9uRm/R+zkeOHEl3W7Zv3z7Ve1Kb91RhxKOlCihnZ2cUK1YMNjY26NatG4yMjLBv3z7Nt+zw8HCcPHkSPXr0QHR0NMLCwhAWFoY3b97AxcUF9+/f1xxdtXv3btSuXTvdbzgymQwAsHPnTlStWhVVqlTRrCssLAwtW7YEAJw6dSrDrG5ubkhKSsKePXs0bUePHkVERATc3NwAAEII7N69G66urhBCpHoOFxcXREZGwt/fP9V63d3dYWBg8NltFR0dDQAwMTHJsE/KY1FRUanau3TpglKlSmnu169fH05OTjh06BAA7bZzikGDBkFHRydV24evIykpCW/evEGFChVgbm6e5nVry8PDI9U3xCZNmgAAHj58CAD4999/8ebNGwwaNAi6uv8b7O3Tp0+qkcCMpGyzT23f9Pzwww+p7jdp0gRv3rxJ9TP4cLtERkYiLCwMzZo1w8OHDxEZGZlq+XLlymlGAT+UmXUcO3YM0dHRmDBhAvT19VMtn/I78Cna/n40a9YM1apV++x6zc3NcenSpVRHA2XV69ev8ffff2PAgAEoU6ZMqsc+9xrfvHkDABm+HwIDA1GsWDEUK1YMVapUwfz589GpU6c0h6FHR0d/9n3y8e9iVFSU1u+tlKyfO91AVt+7mZXez7lly5awtLSEj4+Ppu3t27c4duyY5u8h8GV/cwsD7pYqoJYtW4ZKlSohMjIS69evx99//w2lUql5/MGDBxBCYOrUqZg6dWq663j16hVKlSqFoKAgdO3a9ZPPd//+fdy5cwfFihXLcF0ZqV27NqpUqQIfHx8MHDgQwPtdUpaWlppf1NevXyMiIgKrV6/G6tWrM/Uc5cqV+2TmFCl/uKKjo1MNkX8oowKoYsWKafpWqlQJvr6+ALTbzp/KHRcXBy8vL2zYsAEvXrxIdWj6xx/i2vr4gyzlA+rt27cAoDlnSYUKFVL109XVzXB3yYdMTU0B/G8bZkeulHWeP38enp6euHjxIt69e5eqf2RkJMzMzDT3M3o/ZGYdQUFBAIAaNWpo9RpSaPv7kdn37rx58+Du7g4bGxs4Ojqiffv26NevH+zs7LTOmFLMZvU1AsjwlAm2trZYs2YN1Go1goKCMGvWLLx+/TpNoWhiYvLZguPj30VTU1NNdm2zfq5oy+p7N7PS+znr6uqia9eu2LZtGxISEqBUKrFnzx4kJSWlKm6+5G9uYcDipoCqX78+6tatC+D96ELjxo3Ru3dv3L17F8bGxprzS4wZMybdb7NA2g+zT1Gr1ahZsyYWLlyY7uM2NjafXN7NzQ2zZs1CWFgYTExMsG/fPvTq1UszUpCS99tvv00zNydFrVq1Ut3PzKgN8H5Oyp9//on//vsPTZs2TbfPf//9BwCZ+jb9oaxs5/RyjxgxAhs2bMBPP/2EBg0awMzMDDKZDD179szwXCGZ9fEoUYqMPqi0VaVKFQDAjRs3YG9vn+nlPpcrKCgIrVq1QpUqVbBw4ULY2NhAoVDg0KFDWLRoUZrtkt521XYdWaXt70dm37s9evRAkyZNsHfvXhw9ehTz58/H3LlzsWfPHrRr1+6Lc2dW0aJFAfyvIP6YkZFRqrlqjRo1goODAyZNmoTff/9d0161alUEBATg6dOnaYrbFB//LlapUgXXrl3Ds2fPPvt35kNv375N98vJh7R972ZULKlUqnTbM/o59+zZE6tWrcLhw4fRpUsX+Pr6okqVKqhdu7amz5f+zS3oWNwUAjo6OvDy8kKLFi2wdOlSTJgwQfPNTk9PL9UfnfSUL18eN2/e/Gyf69evo1WrVpkapv+Ym5sbZsyYgd27d8PKygpRUVGaiXMAUKxYMZiYmEClUn02r7Y6duwILy8vbNq0Kd3iRqVSYdu2bbCwsECjRo1SPXb//v00/e/du6cZ0dBmO3/Krl274O7ujgULFmja4uPjERERkapfVrb956SckO3Bgwdo0aKFpj05ORmPHz9OU1R+rF27dtDR0cGWLVuydWLm/v37kZCQgH379qX6INRmOD6z6yhfvjwA4ObNm58s+jPa/l/6+/Ep1tbWGDp0KIYOHYpXr17BwcEBs2bN0hQ3mX2+lPfq537X05NSBDx69ChT/WvVqoVvv/0Wq1atwpgxYzTbvmPHjti+fTs2bdqEKVOmpFkuKioKf/31F6pUqaL5Obi6umL79u3YsmULJk6cmKnnT05OxrNnz9CpU6dP9tP2vWthYZHmdxKA1mdsbtq0KaytreHj44PGjRvj5MmTmDx5cqo+OfmeKgg456aQaN68OerXr4/FixcjPj4exYsXR/PmzbFq1SoEBwen6f/69WvN/7t27Yrr169j7969afqlfIvu0aMHXrx4gTVr1qTpExcXpznqJyNVq1ZFzZo14ePjAx8fH1hbW6cqNHR0dNC1a1fs3r073T++H+bVVsOGDeHs7IwNGzakewbUyZMn4969exg3blyab1p//vlnqjkzly9fxqVLlzQfLNps50/R0dFJM5Lyxx9/pPlGaGRkBADp/oHNqrp166Jo0aJYs2YNkpOTNe1bt27N8Jv6h2xsbDBo0CAcPXoUf/zxR5rH1Wo1FixYgOfPn2uVK2Vk5+NddBs2bMj2dbRp0wYmJibw8vJCfHx8qsc+XNbIyCjd3YRf+vuRHpVKlea5ihcvjpIlSyIhIeGzmT5WrFgxNG3aFOvXr8fTp09TPfa5UbxSpUrBxsZGq7P1jhs3DklJSalGHrp164Zq1aphzpw5adalVqsxZMgQvH37Fp6enqmWqVmzJmbNmoWLFy+meZ7o6Og0hcHt27cRHx+Phg0bfjKjtu/d8uXLIzIyUjO6BADBwcHp/u38FLlcjm7dumH//v3YvHkzkpOTU+2SAnLmPVWQcOSmEBk7diy6d+8Ob29v/PDDD1i2bBkaN26MmjVrYtCgQbCzs0NoaCguXryI58+fa05PPnbsWOzatQvdu3fHgAED4OjoiPDwcOzbtw8rV65E7dq10bdvX/j6+uKHH37AqVOn0KhRI6hUKgQGBsLX1xdHjhzR7CbLiJubG6ZNmwZ9fX0MHDgQcnnq2nvOnDk4deoUnJycMGjQIFSrVg3h4eHw9/fH8ePHER4enuVts2nTJrRq1QqdO3dG79690aRJEyQkJGDPnj04ffo03NzcMHbs2DTLVahQAY0bN8aQIUOQkJCAxYsXo2jRohg3bpymT2a386d07NgRmzdvhpmZGapVq4aLFy/i+PHjmt0BKezt7aGjo4O5c+ciMjISSqUSLVu2RPHixbO8bRQKBaZPn44RI0agZcuW6NGjBx4/fgxvb2+UL18+U98aFyxYgKCgIPz444/Ys2cPOnbsCAsLCzx9+hQ7d+5EYGBgqpG6zGjTpg0UCgVcXV3x/fffIyYmBmvWrEHx4sXTLSS/ZB2mpqZYtGgRvvvuO9SrVw+9e/eGhYUFrl+/jnfv3mHjxo0AAEdHR/j4+GD06NGoV68ejI2N4erqmi2/Hx+Ljo5G6dKl0a1bN80lB44fP44rV66kGuHLKFN6fv/9dzRu3BgODg4YPHgwypUrh8ePH+PgwYMICAj4ZJ7OnTtj7969mZrLArzfrdS+fXusXbsWU6dORdGiRaFQKLBr1y60atUKjRs3hoeHB+rWrYuIiAhs27YN/v7++Pnnn1O9V/T09LBnzx44OzujadOm6NGjBxo1agQ9PT3cunVLM+r64aHsx44dg6GhIVq3bv3ZnNq8d3v27Inx48fj66+/xo8//oh3795hxYoVqFSpktYT/93c3PDHH3/A09MTNWvWTHNKh5x4TxUouX+AFuWkjE7iJ8T7M2CWL19elC9fXnOocVBQkOjXr58oUaKE0NPTE6VKlRIdO3YUu3btSrXsmzdvxPDhwzWnRS9durRwd3dPdVh2YmKimDt3rqhevbpQKpXCwsJCODo6ihkzZojIyEhNv48PBU9x//59zYnGzp07l+7rCw0NFcOGDRM2NjZCT09PlChRQrRq1UqsXr1a0yflEOedO3dqte2io6PF9OnTRfXq1YWBgYEwMTERjRo1Et7e3mkOhf3wJH4LFiwQNjY2QqlUiiZNmojr16+nWXdmtvOnfnZv374VHh4ewtLSUhgbGwsXFxcRGBiY7rZcs2aNsLOzEzo6Opk6id/H2ymjk7v9/vvvomzZskKpVIr69euL8+fPC0dHR9G2bdtMbN33Z3Ndu3ataNKkiTAzMxN6enqibNmywsPDI9WhthmdoThl+3x44sJ9+/aJWrVqCX19fWFrayvmzp0r1q9fn6Zfykn80pPZdaT0bdiwoTAwMBCmpqaifv36Yvv27ZrHY2JiRO/evYW5uXmak/hl9vcD/39yt/Tgg0PBExISxNixY0Xt2rWFiYmJMDIyErVr105zAsKMMmX0c75586b4+uuvhbm5udDX1xeVK1cWU6dOTTfPh/z9/QWANIcmZ3QSPyGEOH36dJrD24UQ4tWrV2L06NGiQoUKQqlUCnNzc+Hs7Kw5/Ds9b9++FdOmTRM1a9YUhoaGQl9fX9SoUUNMnDhRBAcHp+rr5OQkvv3228++phSZfe8KIcTRo0dFjRo1hEKhEJUrVxZbtmz55En8MqJWq4WNjY0AIGbOnJlun8y+pwojmRD55IqARHnI48ePUa5cOcyfPx9jxoyROo4k1Go1ihUrhm+++SbdoXEqfFq1aoWSJUti8+bNUkfJUEBAABwcHODv76/VBHfKXzjnhog+Kz4+Ps28i02bNiE8PBzNmzeXJhTlObNnz4aPj4/WE2hz05w5c9CtWzcWNgUc59wQ0Wf9888/GDVqFLp3746iRYvC398f69atQ40aNdC9e3ep41Ee4eTkhMTERKljfNKOHTukjkC5gMUNEX2Wra0tbGxs8PvvvyM8PBxFihRBv379MGfOnAyv2UVEJBXOuSEiIqIChXNuiIiIqEBhcUNEREQFSqGbc6NWq/Hy5UuYmJjwlNVERET5hBAC0dHRKFmyZJqTvH6s0BU3L1++LPQXFCMiIsqvnj17htKlS3+yT6ErbkxMTAC83zgpl7MnIiKivC0qKgo2Njaaz/FPKXTFTcquKFNTUxY3RERE+UxmppRwQjEREREVKCxuiIiIqEBhcUNEREQFCosbIiIiKlBY3BAREVGBwuKGiIiIChQWN0RERFSgsLghIiKiAoXFDRERERUoLG6IiIioQJG0uPn777/h6uqKkiVLQiaT4c8///zsMqdPn4aDgwOUSiUqVKgAb2/vHM9JRERE+YekxU1sbCxq166NZcuWZar/o0eP0KFDB7Ro0QIBAQH46aef8N133+HIkSM5nJSIiIjyC0kvnNmuXTu0a9cu0/1XrlyJcuXKYcGCBQCAqlWr4ty5c1i0aBFcXFxyKiYREX2JuGBAnSh1CspNciVgUEKyp89XVwW/ePEinJ2dU7W5uLjgp59+ynCZhIQEJCQkaO5HRUXlVDwiIvrYjV+AG55Sp6DcZtkAaHNBsqfPV8VNSEgIrKysUrVZWVkhKioKcXFxMDAwSLOMl5cXZsyYkVsRiYjoQ2+uvP9XpgvI89VHDmkhLMoQaiFDcbPY9w1yhaR5Cvw7beLEiRg9erTmflRUFGxsbCRMRERUCNVfCZQfKHUKygF///0MvUYfRNWqRXDkSDfo6Eh/IHa+Km5KlCiB0NDQVG2hoaEwNTVNd9QGAJRKJZRKZW7EIyIiKjTUagEvr0uYNu081GoBU1MFXr16B2trY6mj5a/ipkGDBjh06FCqtmPHjqFBgwYSJSIiIip8QkNj0bfvIRw79gQA0K9fNSxb5gxjY2l3R6WQdOwoJiYGAQEBCAgIAPD+UO+AgAA8ffoUwPtdSv369dP0/+GHH/Dw4UOMGzcOgYGBWL58OXx9fTFq1Cgp4hMRERU6J08+hb39Jhw79gSGhrrw9m6LjRvb55nCBpB45Obff/9FixYtNPdT5sa4u7vD29sbwcHBmkIHAMqVK4eDBw9i1KhRWLJkCUqXLo21a9fyMHAiIqJckJysxvDhxxESEovq1YvC19cV1apZSh0rDZkQQkgdIjdFRUXBzMwMkZGRMDU1lToOEVHBdtoVeHkAcFrLCcUFxPXrr7By5XUsWNAchoZ6ufa82nx+Sz+lmYiIiPKso0cfY82a/zT3a9cujhUrWudqYaOtfDWhmIiIiHJHcrIanp7n4eV1Cbq6cjg6WsHBwerzC+YBLG5IWvGvgVuzgaQIqZMQUU6ICJA6AWXB8+fR6NXrAM6dewEAGDiwJqpVKypxqsxjcUPSerwFuLtY6hRElNMUFlInoEw6dOgh+vU7jDdv4mBiosDatW3Qo0cVqWNphcUNSUsV9/7fInWBMt2kzUJEOUNZHCjlKnUKyoTJk89i9uxLAAAHByv4+rqifHlzaUNlAYsbyhssagPVxkudgoioUCtSRB8AMGJEHcyf3wxKZf4sE/JnaiIiIsoWsbGJMDJ6fwK+0aPrwsnJGo0bl5Y41ZfhoeBERESFUGKiCj/9dBJ1625BTEwiAEAmk+X7wgZgcUNERFToPHwYgUaNtmHJEn8EBoZj//4gqSNlK+6WIiIiKkR2776HAQP8EBWVCAsLfWzc2A6uruWljpWtWNwQEREVAvHxyRgz5jSWLQsAADRsWBLbt3dEmTIF71JE3C1FRERUCIwde0ZT2IwfXx+nT7sVyMIG4MgNERFRoTB58lc4ffoZ5s9vhrZty0kdJ0dx5IaIiKgAiotLwrZtdzT3S5QwwvXr7gW+sAE4ckNERFTgBAa+QY8e+3HjRhh0dWWayyfI5TKJk+UOFjdEREQFyKZNtzBkyDG8e5eM4sUNUaSIgdSRch2LGyIiogIgNjYRI0acxIYNNwEALVuWwZYt7WFtbSxxstzH4oaIiCifu3UrDD167Mft228gl8vg6dkAkyd/BR2dwjm1lsUNERFRPhcUFIHbt9/A2toI27Z1QPPmZaSOJCkWN0RERPmQEAIy2fsJwp06VcDatS5wdbVD8eJGEieTXuEcryIiIsrHrl9/hcaNt+PZsyhN28CBNVnY/D8WN0RERPmEEAKrVl2Hk9NWXLjwEj//fFrqSHkSd0sRERHlA1FRCRg8+Ch8fO4CADp0sMPy5c4Sp8qbWNwQERHlcf7+oXBz248HDyKgqyuHl1cTjB5dt9CclE9bLG6IiIjysFOnnqJt291ITFShTBkT+Pi44quvSkodK09jcUNERJSHffWVNSpXtoCdnTnWr3cplGcc1haLGyIiojzm1q0wVKlSBDo6chgY6OHUKTcUKaKvOfSbPo1HSxEREeURQggsWvQv6tTZBC+vS5r2okUNWNhogSM3REREeUB4eBz69/fD/v1BAICbN8NSnaiPMo/FDRERkcQuXHiBnj0P4NmzaCgUOli0qDmGDLFnYZNFLG6IiIgkolYL/PbbFUyadBYqlUCFCubw9XVFnTpWUkfL11jcEBERSSQoKALTpp2HSiXQq1cVrFrVBiYmCqlj5XssboiIiCRSsaIFli5tBSGA776ryd1Q2YTFDX05dTIQ9yJryya+zd4sRER5mFotMGfOJTg7l0X9+tYAgO++qyVxqoKHxQ19GSEAv7pAxHWpkxAR5WmhobHo2/cQjh17gjVr/sPNm/1hZMRdUDmBxQ19GXXS/wobuRLIypCqjgFQsmP25iIiykNOnnyKPn0OIiQkFgYGuvD0bMjCJgexuKHs800ooDCTOgURUZ6hUqnx668X8csvFyEEUL16Ufj6uqJaNUupoxVoLG6IiIhyQFRUAjp3/hOnTz8DAAwYUAN//NEKhoZ6Eicr+FjcEBER5QBjYwWMjPRgZKSHlStb49tvq0kdqdBgcUNERJRNkpPVSEpSwcBAD3K5DBs3tkNYWBwqVy4idbRChRfOJCIiygbPn0ejZUtf/PDDMU1b0aIGLGwkwOKGiIjoCx069BD29ptw9uxz7N37AI8fR0odqVBjcUNERJRFSUkqjBt3Bh067MGbN3FwcLCCv39f2NryyFEpcc4NERFRFjx9GoWePQ/g4sWXAIARI+pg/vxmUCr50So1/gSIiIi0pFYLtG27C3fuhMPMTIn1613wzTeVpI5F/4+7pYiIiLQkl8uwZElLfPWVNa5d68vCJo/hyA0REVEmPHwYgaCgCLRubQsAaN3aFq1alYVczit55zUcuSEiIvqM3bvvoU6dTejWbR+CgiI07Sxs8iYWN0RERBmIj0/G8OHH0a3bPkRFJaJ6dUvo6fGjM6/jbikiIqJ03L//Fm5u+3Ht2isAwLhx9TBzZmPo6elInIw+h8UNERHRR3bsCMTgwUcRHZ2IokUNsGlTO7Rvbyd1LMokFjdEREQfuXQpGNHRiWjSpDS2beuA0qVNpI5EWmBxQ0REBEAIAZns/QThuXObokIFc3z/fW3o6nKOTX7DnxgRERV6W7bcRocOe5CcrAYAKBQ6GDasDgubfIo/NSIiKrRiYxMxYIAf+vY9hMOHH2HDhptSR6JswN1SRERUKN26FYYePfbj9u03kMkAT8+GGDCghtSxKBtIPnKzbNky2NraQl9fH05OTrh8+fIn+y9evBiVK1eGgYEBbGxsMGrUKMTHx+dSWiIiyu+EENiw4Qbq1duC27ffoEQJI5w40QOeng2hoyP5xyJlA0l/ij4+Phg9ejQ8PT3h7++P2rVrw8XFBa9evUq3/7Zt2zBhwgR4enrizp07WLduHXx8fDBp0qRcTk5ERPnVjBkXMGDAEcTFJaN167K4fr0fWrQoI3UsykaSFjcLFy7EoEGD4OHhgWrVqmHlypUwNDTE+vXr0+1/4cIFNGrUCL1794atrS3atGmDXr16fXa0h4iIKIWbWxWYmiowa1Zj+Pl1Q/HiRlJHomwmWXGTmJiIq1evwtnZ+X9h5HI4Ozvj4sWL6S7TsGFDXL16VVPMPHz4EIcOHUL79u0zfJ6EhARERUWluhERUeEhhEBAwP/2CFStWhSPHg3CpElf8dpQBZRkxU1YWBhUKhWsrKxStVtZWSEkJCTdZXr37o1ffvkFjRs3hp6eHsqXL4/mzZt/creUl5cXzMzMNDcbG5tsfR1ERJR3RUUloHfvg3B03IyzZ59r2osUMZAwFeW0fDVz6vTp05g9ezaWL18Of39/7NmzBwcPHsSvv/6a4TITJ05EZGSk5vbs2bNcTExERFK5di0Ujo6bsWNHIGQy4M6dN1JHolwi2aHglpaW0NHRQWhoaKr20NBQlChRIt1lpk6dir59++K7774DANSsWROxsbEYPHgwJk+eDLk8ba2mVCqhVCqz/wUQEVGeJITA8uUBGD36NBITVShTxgQ7driiQYOSUkejXCLZyI1CoYCjoyNOnDihaVOr1Thx4gQaNGiQ7jLv3r1LU8Do6Ly/OqsQIufCEhFRvhAREY/u3fdh+PATSExUoVOn8rh2rR8Lm0JG0pP4jR49Gu7u7qhbty7q16+PxYsXIzY2Fh4eHgCAfv36oVSpUvDy8gIAuLq6YuHChahTpw6cnJzw4MEDTJ06Fa6urpoih4iICq8//3yA3bvvQ09PjnnzmmHkSAfN9aKo8JC0uHFzc8Pr168xbdo0hISEwN7eHn5+fppJxk+fPk01UjNlyhTIZDJMmTIFL168QLFixeDq6opZs2ZJ9RKIiCgPcXevjv/+e41evaqgXj1rqeOQRGSikO3PiYqKgpmZGSIjI2Fqaip1nPxPlQj4/P+cpm4RgMJM0jhEVLiEh8dhypRz8PJqCjMzzq8syLT5/Oa1pYiIKF+6ePElevbcj6dPoxEZmYitWztIHYnyiHx1KDgREZFaLTB//mU0bboDT59Go3x5c/z8c12pY1EewpEbIiLKN8LC3sHd/TAOHXoEAHBzq4zVq9vA1JS7pOh/WNwQEVG+EBDwCh077sGLFzFQKnXw++8tMWhQLR4NRWmwuCEionyhdGljAEDlykXg6+uKWrWKSZyI8ioWN0RElGdFRSVodjlZWhriyJFuKFvWFMbGComTUV7GCcVERJQnnTr1FJUrr8fGjTc1bdWrW7Kwoc9icUNERHmKSqXGjBkX4Oy8EyEhsVi2LABqdaE6JRt9Ie6WovfiQgF1vPbLqRKzPwsRFVrBwTH49ttDOHnyKQDAw6MG/vijJeRyThqmzGNxU9jFhQJXhgDP90qdhIgKuWPHHuPbbw/h1at3MDLSw4oVzujbt7rUsSgfYnFTmD3xBf4dCiS8ASADdL7gPBFWLQE9Xs6CiLLm4cMItGu3GyqVQM2alvD1dUWVKkWljkX5FIubwig+DPh3GPDU9/1989pAg42ARW1pcxFRoWVnZ47x4+vjzZt4LFrUHAYGelJHonyMxU1h8/wv4PJgIP4VINMBqk8Cqk8BdHj0ARHlrsOHH6Jy5SKwszMHAMyc2Zgn5KNswaOlCovEt8CFfsDfXd4XNmbVgDb/ALV+YWFDRLkqKUmFcePOoH37PejZ8wASE1UAwMKGsg1HbgqDl4eBS98BcS8BmRyoMgaoNQPQ0Zc6GREVMk+fRqFnzwO4ePElAKB+/RIQgod5U/ZicVOQJUUB/qOBoHXv75tUBL7aCBRrIG0uIiqU9u17gP79/fD2bTzMzJRYt84FXbtWkjoWFUAsbgqqkBPAPwOAd08ByIDKI4HaswBdQ6mTEVEhk5iowoQJf2PRoqsAgHr1SmDHjo6auTZE2Y3FTUGTFAMEjAfuL39/36gc0MAbKN5U0lhEVHgJIfD3388BAD/95Ii5c5tCodCROBUVZCxuCpJXfwP/eAAxD9/frzgEsJ8H6BlLm4uICiUhBGQyGZRKXfj6uuLGjTB07lxB6lhUCLC4KQiEAK5PBG7PAyAAQxvgq/VACWepkxFRIZSQkIwxY87A3FyJX39tDOD9eWy4G4pyC4ubgiAuGLg99/3/7QYADgsBhZm0mYioUHrw4C3c3A7A3z8UcrkM7u7VUaGChdSxqJBhcVMQiKT3/+oYAF+tkzYLERVavr6B+O67o4iOTkTRogbYuLEtCxuSBIsbIiL6InFxSRg16jRWrboOAGjcuBS2b++I0qVNJE5GhRWLGyIiyjIhBJydd+LChZeQyYCJE50wY0Yj6OryBPgkHRY3RESUZTKZDIMG1cL9+2+xZUsHtGljK3UkIhY3RESknXfvkvDkSRSqVi0KAOjfvwY6d64ACwte0oXyBo4bEhFRpt2+HYb69begTZtdePMmTtPOwobyEhY3RESUKd7eN1G37hbcuvUGyclqPH4cKXUkonRxtxQREX1STEwihg07jk2bbgMAnJ3LYsuW9rCyMpI4GVH6WNzkJXEhgDpB++XePc/+LEREAG7ceI0ePfYjMDAccrkMv/zSCBMnOkEul0kdjShDLG7yitvz3l/wkogoD5k79zICA8NRsqQxtm/vgKZNbaSORPRZLG7yijdX3v8r0wXkWfyxlO2VfXmIiAAsW+YMAwNdzJ7dBMWKGUodhyhTWNzkNY5LgEpDpU5BRIXUtWuh2LbtDubNawaZTAYzMyXWrHGROhaRVr6ouImPj4e+Pg//IyLK74QQWLEiAKNGnUZiogrVqhWFh0dNqWMRZYnWh4Kr1Wr8+uuvKFWqFIyNjfHw4UMAwNSpU7FuHS/aSESU30RGJqBHj/0YNuwEEhNVcHUtj86dK0gdiyjLtC5uZs6cCW9vb8ybNw8KhULTXqNGDaxduzZbwxERUc66ciUYdepswq5d96CnJ8fChc3x119dUKSIgdTRiLJM6+Jm06ZNWL16Nfr06QMdHR1Ne+3atREYGJit4YiIKOesX38DjRptx6NHkbC1NcW5c70walRdyGQ8zJvyN63n3Lx48QIVKqQdrlSr1UhKSsqWUERElPMqVDCHSiXwzTcVsW6dC8zNOYeSCgati5tq1arh7NmzKFu2bKr2Xbt2oU6dOtkWjIiIsl9ERLymiGna1AaXLvWBo6MVR2uoQNG6uJk2bRrc3d3x4sULqNVq7NmzB3fv3sWmTZtw4MCBnMhIRERfSK0WWLjwX8ya9Q8uXuyNKlXeX9G7bt0SEicjyn5az7np3Lkz9u/fj+PHj8PIyAjTpk3DnTt3sH//frRu3TonMhIR0RcIC3uHTp32YuzYM4iISMDmzbeljkSUo7J0npsmTZrg2LFj2Z2FiIiy2blzz9Gr10E8fx4NpVIHS5a0xODBtaSORZSjtB65sbOzw5s3b9K0R0REwM7OLltCERHRl1GrBby8LqF5cx88fx6NSpUscOlSH3z/fW3Or6ECT+uRm8ePH0OlUqVpT0hIwIsXL7IlFBERfRlv75uYNOksAODbb6thxQpnGBsrPrMUUcGQ6eJm3759mv8fOXIEZmZmmvsqlQonTpyAra1ttoYjIqKs6devOnbsCETPnlXg4VGDozVUqGS6uOnSpQsAQCaTwd3dPdVjenp6sLW1xYIFC7I1HBERZY5Kpca6dTfQv38NKBQ60NWV48iRbixqqFDKdHGjVqsBAOXKlcOVK1dgaWmZY6GIiCjzQkJi0afPQZw8+RSBgeFYuLAFALCwoUJL6zk3jx49yokcRESUBcePP8G33x5EaOg7GBrqok6d4lJHIpJclg4Fj42NxZkzZ/D06VMkJiameuzHH3/MlmBERJSx5GQ1Zsy4gFmz/oEQQM2alvD1ddWcnI+oMNO6uLl27Rrat2+Pd+/eITY2FkWKFEFYWBgMDQ1RvHhxFjdERDnsxYto9O59EH///RwAMGhQLSxZ0gIGBnoSJyPKG7Q+z82oUaPg6uqKt2/fwsDAAP/88w+ePHkCR0dH/PbbbzmRkYiIPhAXl4xr117B2FgP27Z1wOrVbVjYEH1A65GbgIAArFq1CnK5HDo6OkhISICdnR3mzZsHd3d3fPPNNzmRk4ioUBNCaCYIV6hgAV9fV5Qvb46KFS0kTkaU92g9cqOnpwe5/P1ixYsXx9OnTwEAZmZmePbsWfamIyIiPHsWhWbNfHD8+BNNW9u25VjYEGVA65GbOnXq4MqVK6hYsSKaNWuGadOmISwsDJs3b0aNGjVyIiMRUaG1f38Q+vc/jPDweAwbdhy3b3tAR0fr76VEhYrWvyGzZ8+GtbU1AGDWrFmwsLDAkCFD8Pr1a6xatSrbAxIRFUaJiSr8/PMpdOq0F+Hh8ahb1wqHD3dlYUOUCVqP3NStW1fz/+LFi8PPzy9bAxERFXaPH0fCzW0/Ll8OAQCMHOmAuXObQqnM0tk7iAqdbPsK4O/vj44dO2q93LJly2Brawt9fX04OTnh8uXLn+wfERGBYcOGwdraGkqlEpUqVcKhQ4eyGpuIKE959iwKdepswuXLITA3V2Lv3s5YvLglCxsiLWhV3Bw5cgRjxozBpEmT8PDhQwBAYGAgunTpgnr16mku0ZBZPj4+GD16NDw9PeHv74/atWvDxcUFr169Srd/YmIiWrdujcePH2PXrl24e/cu1qxZg1KlSmn1vEREeVXp0iZwdS2Pr76yRkBAP3TpUlHqSET5Tqa/Cqxbtw6DBg1CkSJF8PbtW6xduxYLFy7EiBEj4Obmhps3b6Jq1apaPfnChQsxaNAgeHh4AABWrlyJgwcPYv369ZgwYUKa/uvXr0d4eDguXLgAPb3353TglciJKL8LCoqAubkSRYsaQCaTYeXK1tDTk0NPT0fqaET5UqZHbpYsWYK5c+ciLCwMvr6+CAsLw/Lly3Hjxg2sXLlS68ImMTERV69ehbOz8//CyOVwdnbGxYsX011m3759aNCgAYYNGwYrKyvUqFEDs2fPhkqlyvB5EhISEBUVlepGRJRX+PoGok6dTfDw8IMQAgBgaKjHwoboC2S6uAkKCkL37t0BAN988w10dXUxf/58lC5dOktPHBYWBpVKBSsrq1TtVlZWCAkJSXeZhw8fYteuXVCpVDh06BCmTp2KBQsWYObMmRk+j5eXF8zMzDQ3GxubLOUlIspO8fHJGDLkGNzcDiA6OhHh4fGIikr8/IJE9FmZLm7i4uJgaGgIAJDJZFAqlZpDwnOLWq1G8eLFsXr1ajg6OsLNzQ2TJ0/GypUrM1xm4sSJiIyM1Nx4okEiktq9e+H46qutWLnyOgBg4kQnnD7tBjMzpcTJiAoGrabfr127FsbGxgCA5ORkeHt7w9LSMlWfzF4409LSEjo6OggNDU3VHhoaihIlSqS7jLW1NfT09KCj87/h2qpVqyIkJASJiYlQKBRpllEqlVAq+QeDiPKGrVtv4/vvjyE2NgnFihlg8+b2cHEpJ3UsogIl08VNmTJlsGbNGs39EiVKYPPmzan6yGSyTBc3CoUCjo6OOHHiBLp06QLg/cjMiRMnMHz48HSXadSoEbZt2wa1Wq25BMS9e/dgbW2dbmFDRJSXvHuXhClTziE2NgnNm9tg69YOKFnSWOpYRAVOpoubx48fZ/uTjx49Gu7u7qhbty7q16+PxYsXIzY2VnP0VL9+/VCqVCl4eXkBAIYMGYKlS5di5MiRGDFiBO7fv4/Zs2dnuqAiIpKSoaEefHxccejQQ0yd2oBnGybKIZKeFcrNzQ2vX7/GtGnTEBISAnt7e/j5+WkmGT99+lQzQgMANjY2OHLkCEaNGoVatWqhVKlSGDlyJMaPHy/VSyAi+qSNG29CpRIYMKAmAKB+fWvUr5+78xWJChuZSDn2sJCIioqCmZkZIiMjYWpqKnWc/znbHXi2C6i7DKg0VOo0RPSFYmISMWzYcWzadBtKpQ7++88dlSoVkToWUb6lzec3z+dNRJTNbtx4jR499iMwMBxyuQxTpnyF8uXNpY5FVGiwuCEiyiZCCKxbdwMjRpxEfHwySpY0xrZtHdCsGc+vRZSbWNwQEWUDIQTc3Q9j8+bbAIC2bW2xaVN7FCtmKHEyosInS1P1g4KCMGXKFPTq1UtzkcvDhw/j1q1b2RqOiCi/kMlkqFjRAjo6MsyZ0wQHD3ZlYUMkEa2LmzNnzqBmzZq4dOkS9uzZg5iYGADA9evX4enpme0BiYjyKiEE3r6N19yfNMkJV6/2xfjxTpDLZRImIyrctC5uJkyYgJkzZ+LYsWOpTpzXsmVL/PPPP9kajogor4qMTICb2340b+6DuLgkAICOjhy1axeXOBkRaV3c3LhxA19//XWa9uLFiyMsLCxbQhER5WX//hsCB4dN2LnzHm7ffoPz519KHYmIPqB1cWNubo7g4OA07deuXUOpUqWyJRQRUV4khMDvv/ujYcNtePgwEmXLmuLcuV5wdi4rdTQi+oDWxU3Pnj0xfvx4hISEQCaTQa1W4/z58xgzZgz69euXExmJiCT39m08vvnmL4wceRJJSWp06VIB1671g5MTzzZMlNdoXdzMnj0bVapUgY2NDWJiYlCtWjU0bdoUDRs2xJQpU3IiIxGR5IYOPY4//3wAhUIHv//eEnv2dIaFhb7UsYgoHVqf50ahUGDNmjWYOnUqbt68iZiYGNSpUwcVK1bMiXxERHnC3LlNERQUgRUrnOHoWELqOET0CVoXN+fOnUPjxo1RpkwZlClTJicyERFJ7s2bOOzfH4T+/WsAAMqUMcWlS30gk/EQb6K8TuvdUi1btkS5cuUwadIk3L59OycyERFJ6vz5F7C33wQPDz/s3x+kaWdhQ5Q/aF3cvHz5Ej///DPOnDmDGjVqwN7eHvPnz8fz589zIh8RUa5RqwXmzLmEZs124PnzaFSsaAEbGxOpYxGRlrQubiwtLTF8+HCcP38eQUFB6N69OzZu3AhbW1u0bNkyJzISEeW4V69i0b79bkyceBYqlUDv3lVx9Wpf2NvzpHxE+c0XXTizXLlymDBhAmrXro2pU6fizJkz2ZWLiCjXnDnzDL16HUBwcCz09XWxdGkrDBhQg7uhiPKpLF04EwDOnz+PoUOHwtraGr1790aNGjVw8ODB7MxGRJQrgoNjERwci6pVi+DKlT4YOLAmCxuifEzrkZuJEydix44dePnyJVq3bo0lS5agc+fOMDTk1W+JKP8QQmgKmJ49qyAxUYWuXSvCyEjxmSWJKK/TeuTm77//xtixY/HixQscOHAAvXr1YmFDRPnKiRNP4OCwGSEhsZq2fv2qs7AhKiC0Hrk5f/58TuQgIspxKpUaM2ZcwMyZ/0AIYMaMC1ixorXUsYgom2WquNm3bx/atWsHPT097Nu375N9O3XqlC3BiIiy08uXMejd+wDOnHl/2orvvquJBQuaSxuKiHJEpoqbLl26ICQkBMWLF0eXLl0y7CeTyaBSqbIrGxFRtjhy5BG+/fYQwsLiYGysh1Wr2qB376pSxyKiHJKp4katVqf7fyKivG7nzrvo0WM/AKB27WLw9XVFpUpFJE5FRDlJ6wnFmzZtQkJCQpr2xMREbNq0KVtCERFll7Zty6FSJQsMHWqPf/7pw8KGqBDQurjx8PBAZGRkmvbo6Gh4eHhkSygioi/xzz8vIYQAAJiYKHDlyrdYtswZ+vpfdN5SIsontC5uPjw3xIeeP38OMzOzbAlFRJQViYkqjBlzGg0abMPixVc17aamSglTEVFuy/TXmDp16kAmk0Emk6FVq1bQ1f3foiqVCo8ePULbtm1zJCQR0ec8fhyJnj0P4NKlYADAixcxEiciIqlkurhJOUoqICAALi4uMDY21jymUChga2uLrl27ZntAIqLP+fPP+/Dw8ENERALMzZXYsKEtunSpKHUsIpJIposbT09PAICtrS3c3Nygr6+fY6GIiDIjISEZ48b9jd9/9wcAODlZY8eOjrC15S5yosJM6zk37u7uLGyIKE+4ffsNli8PAAD8/HNd/P13TxY2RJS5kZsiRYrg3r17sLS0hIWFxSevlhseHp5t4YiIPqVOHSv88UdLlC5tgo4dy0sdh4jyiEwVN4sWLYKJiYnm/58qboiIckp8fDLGj/8bAwfWRK1axQAAP/xgL20oIspzMlXcuLu7a/7fv3//nMpCRJShe/fC0aPHfly//hpHjz7GjRv9oaur9Z51IioEtP7L4O/vjxs3bmju//XXX+jSpQsmTZqExMTEbA1HRAQA27bdgaPjZly//hrFihlg8eIWLGyIKENa/3X4/vvvce/ePQDAw4cP4ebmBkNDQ+zcuRPjxo3L9oBEVHi9e5eEQYOOoE+fg4iJSUKzZqUREOAOF5dyUkcjojxM6+Lm3r17sLe3BwDs3LkTzZo1w7Zt2+Dt7Y3du3dndz4iKqRCQmLh5LQVa9fegEwGTJvWAMeP90DJksafX5iICjWtL7QihNBcGfz48ePo2LEjAMDGxgZhYWHZm46ICq1ixQxQvLghrKwMsXVrB7RqVVbqSESUT2hd3NStWxczZ86Es7Mzzpw5gxUrVgAAHj16BCsrq2wPSESFR2xsInR05NDX14WOjhxbt3YAAJQoYSRxMiLKT7TeLbV48WL4+/tj+PDhmDx5MipUqAAA2LVrFxo2bJjtAYmocLh58zXq1duCUaNOadpKlDBiYUNEWtN65KZWrVqpjpZKMX/+fOjo6GRLKCIqPIQQWL/+JoYPP4H4+GRERiZi5szGKFrUQOpoRJRPaV3cpLh69Sru3LkDAKhWrRocHByyLRQRFQ7R0YkYMuQYtm59/7fExcUWmze3Z2FDRF9E6+Lm1atXcHNzw5kzZ2Bubg4AiIiIQIsWLbBjxw4UK1YsuzMSUQF0/for9OixH/fuvYWOjgwzZzbGuHH1IZfzDOhE9GW0nnMzYsQIxMTE4NatWwgPD0d4eDhu3ryJqKgo/PjjjzmRkYgKmISEZLRvvwf37r1F6dImOHOmJyZMcGJhQ0TZQuuRGz8/Pxw/fhxVq1bVtFWrVg3Lli1DmzZtsjUcERVMSqUuVqxwxpo1/8Hbux13QxFRttK6uFGr1dDT00vTrqenpzn/DRHRx65eDcHbtwlwdn5/vppOnSrA1bU8L8RLRNlO691SLVu2xMiRI/Hy5UtN24sXLzBq1Ci0atUqW8MRUf4nhMAff/ijYcPtcHPbj2fPojSPsbAhopyg9cjN0qVL0alTJ9ja2sLGxgYA8OzZM9SoUQNbtmzJ9oD5SvI7IOF1FpeNzd4sRHnA27fxGDjwCPbuvQ8AaNq0HIyNFRKnIqKCTuvixsbGBv7+/jhx4oTmUPCqVavC2dk528PlK/FhwP6KQFKE1EmI8oRLl4LRs+d+PH4cBYVCB7/91gzDh9fhaA0R5TitihsfHx/s27cPiYmJaNWqFUaMGJFTufKfmAf/K2x09LO2DmUxwKp5diUikoQQAosWXcX48X8jOVkNOzsz+Pq6wtGxhNTRiKiQyHRxs2LFCgwbNgwVK1aEgYEB9uzZg6CgIMyfPz8n8+U/xnZApyCpUxBJRiaTITAwHMnJanTvXglr1rjAzEwpdSwiKkQyPaF46dKl8PT0xN27dxEQEICNGzdi+fLlOZmNiPIRtVpo/r9kSQts2dIePj6uLGyIKNdlurh5+PAh3N3dNfd79+6N5ORkBAcH50gwIsof1GqBuXMvoWPHPZoCx8BAD336VOP8GiKSRKZ3SyUkJMDI6H9X55XL5VAoFIiLi8uRYESU971+/Q79+h2Cn99jAMBffz3A119XlDYUERV6Wk0onjp1KgwNDTX3ExMTMWvWLJiZmWnaFi5cmH3piCjP+vvvZ+jV6yBevoyBvr4uli5thS5dKkgdi4go88VN06ZNcffu3VRtDRs2xMOHDzX3OQRNVPCpVGp4eV2Cp+cFqNUCVasWga+vK2rU4EVziShvyHRxc/r06RyMQUT5xdChx7F69X8AgP79q2Pp0lYwMuKJ+Ygo79D68gs5YdmyZbC1tYW+vj6cnJxw+fLlTC23Y8cOyGQydOnSJWcDEpHGkCH2KFJEHxs3tsOGDe1Y2BBRniN5cePj44PRo0fD09MT/v7+qF27NlxcXPDq1atPLvf48WOMGTMGTZo0yaWkRIWTSqXGxYv/u5acvX1xPHkyGP36VZcwFRFRxiQvbhYuXIhBgwbBw8MD1apVw8qVK2FoaIj169dnuIxKpUKfPn0wY8YM2NnZ5WJaosLl5csYtGrli2bNduDKlf+d9oHXhyKivEzS4iYxMRFXr15NdV0quVwOZ2dnXLx4McPlfvnlFxQvXhwDBw7MjZhEhdKRI49gb78RZ848h1Kpg5cveXFXIsoftL5wZnYKCwuDSqWClZVVqnYrKysEBgamu8y5c+ewbt06BAQEZOo5EhISkJCQoLkfFRWV5bxEhUFyshpTp57DnDnv577Vrl0Mvr6uqFSpiMTJiIgyJ0sjN2fPnsW3336LBg0a4MWLFwCAzZs349y5c9ka7mPR0dHo27cv1qxZA0tLy0wt4+XlBTMzM83NxsYmRzMS5WfPnkWheXMfTWEzdKg9/vmnDwsbIspXtC5udu/eDRcXFxgYGODatWuaUZHIyEjMnj1bq3VZWlpCR0cHoaGhqdpDQ0NRokTaKwgHBQXh8ePHcHV1ha6uLnR1dbFp0ybs27cPurq6CApKe8HKiRMnIjIyUnN79uyZVhmJCpM9e+7j/PkXMDVVwNfXFcuWOUNfX9IBXiIirWld3MycORMrV67EmjVroKenp2lv1KgR/P39tVqXQqGAo6MjTpw4oWlTq9U4ceIEGjRokKZ/lSpVcOPGDQQEBGhunTp1QosWLRAQEJDuqIxSqYSpqWmqGxGlb8QIB4wbVw/+/v3QvXtlqeMQEWWJ1l/J7t69i6ZNm6ZpNzMzQ0REhNYBRo8eDXd3d9StWxf169fH4sWLERsbCw8PDwBAv379UKpUKXh5eUFfXx81atRItby5uTkApGknos978iQSU6eex/LlzjA2VkAul2Hu3GZSxyIi+iJaFzclSpTAgwcPYGtrm6r93LlzWTos283NDa9fv8a0adMQEhICe3t7+Pn5aSYZP336FHK55EesExU4f/31AP37H0ZERAKMjfWwfHlrqSMREWULrYubQYMGYeTIkVi/fj1kMhlevnyJixcvYsyYMZg6dWqWQgwfPhzDhw9P97HPXfbB29s7S89JVFglJqowbtwZLFnyfjdy/folMG5cfYlTERFlH62LmwkTJkCtVqNVq1Z49+4dmjZtCqVSiTFjxmDEiBE5kZGIssnDhxFwc9uPf/99P4n/55/rYvbsJlAodCRORkSUfbQubmQyGSZPnoyxY8fiwYMHiImJQbVq1WBsbJwT+Ygom5w+/RSdO/+JqKhEzbWhOnYsL3UsIqJsl+VjPBUKBapVq5adWYgoB1WuXAT6+rqoWbMYtm/vABsbHjlIRAWT1sVNixYtIJPJMnz85MmTXxSIiLJPWNg7WFoaAgCsrY1x5owbypc3h54ed0MRUcGl9WFI9vb2qF27tuZWrVo1JCYmwt/fHzVr1syJjESUBdu334Gd3Vrs2nVX01alSlEWNkRU4Gk9crNo0aJ026dPn46YmJgvDkREXyYuLgkjR57CmjX/AQA2bbqNbt14Qj4iKjyy7QQy3377LdavX59dqyOiLAgMfAMnp61Ys+Y/yGTA1KlfYc+ezlLHIiLKVdl20ZiLFy9CX18/u1ZHRFratOkWhgw5hnfvkmFlZYgtWzrA2bms1LGIiHKd1sXNN998k+q+EALBwcH4999/s3wSPyL6Mv7+oXB3PwwAaNmyDLZu7YASJYwkTkVEJA2tixszM7NU9+VyOSpXroxffvkFbdq0ybZgRJR5Dg5W+PnnujAzU2LSJCfo6PCSJURUeGlV3KhUKnh4eKBmzZqwsLDIqUxE9BlCCGzadAutWpVF6dImAIDffmsubSgiojxCq693Ojo6aNOmTZau/k1E2SM6OhF9+x5C//5+6NXrAJKT1VJHIiLKU7Qeu65RowYePnyYE1mI6DOuX3+FunU3Y+vWO9DRkaFDBzvI5RmfVJOIqDDSuriZOXMmxowZgwMHDiA4OBhRUVGpbkSU/YQQWLXqOpyctuLevbcoXdoEZ870xIQJTixuiIg+kuk5N7/88gt+/vlntG/fHgDQqVOnVJdhEEJAJpNBpVJlf0qiQiw6OhHffXcEvr7vzzTcsaMdvL3boWhRA4mTERHlTZkubmbMmIEffvgBp06dysk8RPQRHR0Zbt9+A11dOebMaYLRo+t+8vpuRESFXaaLGyEEAKBZs2Y5FoaI3hNCQAhALpfB0FAPvr6uiIxMwFdflZQ6GhFRnqfVnBt+WyTKeRER8ejWbR/mzr2saatatSgLGyKiTNLqPDeVKlX6bIETHh7+RYGICrPLl4Ph5rYfjx9H4fDhRxgwoAasrHimYSIibWhV3MyYMSPNGYqJ6MsJIbB48VWMH/83kpLUsLMzg4+PKwsbIqIs0Kq46dmzJ4oXL55TWYgKpfDwOPTv74f9+4MAAN26VcLatS4wM1NKnIyIKH/KdHHD+TZE2S8xUYWvvtqG+/ffQqnUwaJFLfDDD7X5+0ZE9AUyPaE45WgpIso+CoUOfvrJARUrWuCff/pgyBB7FjZERF8o0yM3ajWvX0OUHcLC3uHVq3eoVs0SADBkiD36968BQ0M9iZMRERUMWl9+gYiy7uzZ56hdexNcXfciMjIBwPtdvixsiIiyD4sbolygVgvMmvUPmjf3wcuXMVAodPD69TupYxERFUhaHS1FRNoLDY1F376HcOzYEwCAu3t1LFvWCkZGComTEREVTCxuiHLQyZNP0afPQYSExMLQUBfLlzvD3b2G1LGIiAo0FjdEOWjRon8REhKL6tWLwtfXVTOJmIiIcg7n3BDloA0b2mLMmLq4fPlbFjZERLmExQ1RNjp69DHGjDmtuW9paYj585vzaCgiolzE3VJE2SA5WQ1Pz/Pw8roEIYCGDUvim28qSR2LiKhQYnFD9IWeP49G794HcfbscwDADz/URrt25SRORURUeLG4IfoChw49RL9+h/HmTRxMTBRYu7YNevSoInUsIqJCjcUNURbNnv0PJk8+BwBwdLSCj48rypc3lzYUERFxQjFRVjk6WkEmA0aMqIPz53uxsCEiyiM4ckOkhVevYlG8uBEAwMWlHG7d8kDVqkUlTkVERB/iyA1RJiQmqjBq1ClUrrweDx9GaNpZ2BAR5T0sbog+49GjCDRuvB2LF19FREQCDh9+JHUkIiL6BO6WIvqE3bvvYeDAI4iMTECRIvrw9m4HV9fyUsciIqJPYHFDlI74+GSMGXMay5YFAHh/Ur7t2zuiTBlTaYMREdFncbcUUTp+/91fU9iMH18fp0+7sbAhIsonOHJDlI6RIx1w6tRT/PijA9q1s5M6DhERaYEjN0QA4uKS8NtvV5CcrAYAKJW6OHy4GwsbIqJ8iCM3VOgFBr5Bjx77ceNGGCIiEjBzZmOpIxER0RdgcUOF2ubNtzBkyHHExibBysoQzZvbSB2JiIi+EIsbKpRiYxMxYsRJbNhwEwDQsmUZbN3aASVKGEmcjIiIvhSLGyp07tx5g27d9uH27TeQy2Xw9GyAyZO/go4Op6ARERUELG6o0FGrBR49ioS1tRG2beuA5s3LSB2JiIiyEYsbKhRUKrVmZKZ6dUvs3dsZdeoU11wEk4iICg6Ow1OBd/36K9SqtRHnzj3XtLm4lGNhQ0RUQLG4oQJLCIFVq67DyWkrbt9+g7Fjz0AIIXUsIiLKYdwtRQVSVFQCBg8+Ch+fuwCA9u3LYePGdpDJZBInIyKinMbihgocf/9QuLntx4MHEdDVlcPLqwlGj64LuZyFDRFRYcDihgqUmzdfo0GDbUhMVKFMGRPs2OGKBg1KSh2LiIhyEYsbKlCqV7dEx452SE5WY8OGtihSxEDqSERElMvyxITiZcuWwdbWFvr6+nBycsLly5cz7LtmzRo0adIEFhYWsLCwgLOz8yf7U8H3778hiIxMAADIZDJs2dIef/7ZhYUNEVEhJXlx4+Pjg9GjR8PT0xP+/v6oXbs2XFxc8OrVq3T7nz59Gr169cKpU6dw8eJF2NjYoE2bNnjx4kUuJyepCSGwaNG/aNhwGwYPPqo5EsrAQI8Th4mICjHJi5uFCxdi0KBB8PDwQLVq1bBy5UoYGhpi/fr16fbfunUrhg4dCnt7e1SpUgVr166FWq3GiRMncjk5SSk8PA5duvyJ0aNPIylJDbVaIDFRJXUsIiLKAyQtbhITE3H16lU4Oztr2uRyOZydnXHx4sVMrePdu3dISkpCkSJFciom5TEXL76Evf0m7NsXBIVCB8uWtYKvryuUSk4hIyIiiScUh4WFQaVSwcrKKlW7lZUVAgMDM7WO8ePHo2TJkqkKpA8lJCQgISFBcz8qKirrgUlSarXAb79dwaRJZ6FSCVSoYA5fX1fUqWP1+YWJiKjQkHy31JeYM2cOduzYgb1790JfXz/dPl5eXjAzM9PcbGxscjklZZeIiHgsWeIPlUqgV68q8Pfvx8KGiIjSkLS4sbS0hI6ODkJDQ1O1h4aGokSJEp9c9rfffsOcOXNw9OhR1KpVK8N+EydORGRkpOb27NmzbMlOua9IEQNs394Bq1e3wdatHWBiopA6EhER5UGSFjcKhQKOjo6pJgOnTA5u0KBBhsvNmzcPv/76K/z8/FC3bt1PPodSqYSpqWmqG+UParXArFn/YMuW25q2pk1tMGhQLR4NRUREGZJ8Bubo0aPh7u6OunXron79+li8eDFiY2Ph4eEBAOjXrx9KlSoFLy8vAMDcuXMxbdo0bNu2Dba2tggJCQEAGBsbw9jYWLLXQdkrNDQWffsewrFjT2BoqIsWLWxQqpSJ1LGIiCgfkLy4cXNzw+vXrzFt2jSEhITA3t4efn5+mknGT58+hVz+vwGmFStWIDExEd26dUu1Hk9PT0yfPj03o1MOOXXqKXr3PoiQkFgYGOhi6dJWKFmShSsREWWOTKSc+ayQiIqKgpmZGSIjI7N3F1XYP8DRBoCxHdApKPvWW4ioVGrMnPkPfvnlItRqgerVi8LX1xXVqllKHY2IiCSmzee35CM3RACQnKxG27a7cOLEUwDAwIE18fvvLWFoqCdxMiIiym/y9aHgVHDo6spRr14JGBnpYcuW9li71oWFDRERZQlHbkgyyclqvH0bj2LFDAEAv/zSCN99Vwvly5tLG4yIiPI1jtyQJJ4/j0aLFj7o0GGP5ppQeno6LGyIiOiLsbihXHfo0EPY22/CuXMvEBgYjps3w6SOREREBQiLG8o1SUkqjBt3Bh067MGbN3FwcLCCv39fODjwEgpERJR9OOeGcsWTJ5Ho2fMA/vknGAAwYkQdzJ/fjFfyJiKibMdPFsoV3313FP/8EwwzMyXWr3fBN99UkjoSEREVUNwtRblixQpnODuXxbVrfVnYEBFRjmJxQzni0aMIrF37n+Z+hQoWOHasO8qVM5cuFBERFQrcLUXZbvfuexg48AiiohJga2sGZ+eyUkciIqJChMUNZZv4+GSMGXMay5YFAAAaNCiJihXNJc1ERESFD4sbyhYPHrxFjx77ce3aKwDAuHH1MHNmY+jp6UicjIiIChsWN/TFdu68i4EDjyA6OhFFixpg06Z2aN/eTupYRERUSLG4oS8WE5OI6OhENGlSGtu2dUDp0iZSRyIiokKMxQ1lSXKyGrq67w+269+/BoyNFfj664qaNiIiIqnwk4i0tnnzLdSq5Y03b+IAADKZDN27V2ZhQ0REeQI/jSjTYmMTMWCAH/r1O4w7d8Lx++/+UkciIiJKg7ulKFNu3QpDjx77cfv2G8hkgKdnQ0yZ8pXUsYiIiNJgcUOfJISAt/dNDBt2AnFxyShRwgjbtnVAixZlpI5GRESULhY39EnLlwdg+PATAIDWrcti8+b2sLIykjgVERFRxjjnhj6pT5+qqFDBHLNmNYafXzcWNkRElOdx5IZSEULg+PEncHYuC5lMBnNzfdy40R/6+nyrEBFR/sCRG9KIikpA794H0abNLqxZ878rerOwISKi/ISfWgQAuHYtFD167MeDBxHQ1ZUjLi5Z6khERFmiVquRmJgodQzKAoVCAbn8y8ddWNwUckIILF8egNGjTyMxUYUyZUywY4crGjQoKXU0IiKtJSYm4tGjR1Cr1VJHoSyQy+UoV64cFArFF62HxU0hFhERj+++O4Ldu+8DADp1Ko8NG9qiSBEDiZMREWlPCIHg4GDo6OjAxsYmW0YAKPeo1Wq8fPkSwcHBKFOmDGQyWZbXxeKmELtxIwx79z6Anp4c8+Y1w8iRDl/0ZiIiklJycjLevXuHkiVLwtDQUOo4lAXFihXDy5cvkZycDD09vSyvh8VNIdakSWksXdoKdetaoV49a6njEBF9EZVKBQBfvEuDpJPys1OpVF9U3HDMrhAJD49D794HcPduuKZtyBB7FjZEVKBwBDr/yq6fHUduComLF1+iZ8/9ePo0Gg8eRODSpT78A0BERAUSR24KOLVaYP78y2jadAeePo1G+fLmWLmyNQsbIqI8on///pDJZJDJZNDT00O5cuUwbtw4xMfHp+p34MABNGvWDCYmJjA0NES9evXg7e2d7jp3796N5s2bw8zMDMbGxqhVqxZ++eUXhIeHp9u/oGFxU4CFhb2Dq+sejBv3N5KT1XBzqwx//75wcLCSOhoREX2gbdu2CA4OxsOHD7Fo0SKsWrUKnp6emsf/+OMPdO7cGY0aNcKlS5fw33//oWfPnvjhhx8wZsyYVOuaPHky3NzcUK9ePRw+fBg3b97EggULcP36dWzevDm3X5okuFuqgHrw4C2aN/fBixcx0NfXxZIlLTBoUC2O2BAR5UFKpRIlSpQAANjY2MDZ2RnHjh3D3Llz8ezZM/z888/46aefMHv2bM0yP//8MxQKBX788Ud0794dTk5OuHz5MmbPno3Fixdj5MiRmr62trZo3bo1IiIicvulSYLFTQFVtqwpypY1hbGxAr6+rqhVq5jUkYiIcpcQgOqdNM+tYwhk8cvkzZs3ceHCBZQtWxYAsGvXLiQlJaUZoQGA77//HpMmTcL27dvh5OSErVu3wtjYGEOHDk133ebm5lnKlN+wuClAXr9+BzMzJRQKHejp6WDXrk4wMVHA2JiHRRJRIaR6B/gaS/PcPWIAXaNMdz9w4ACMjY2RnJyMhIQEyOVyLF26FABw7949mJmZwdo67ZGtCoUCdnZ2uHfvHgDg/v37sLOz+6LDqAsCzrkpIE6deopatTZi0qSzmjZra2MWNkRE+UCLFi0QEBCAS5cuwd3dHR4eHujatavW6xFC5EC6/IcjN/mcSqXGzJn/4JdfLkKtFvDze4RffmkEQ8PCXbUTEUHH8P0IilTPrQUjIyNUqFABALB+/XrUrl0b69atw8CBA1GpUiVERkbi5cuXKFky9XX/EhMTERQUhBYtWgAAKlWqhHPnziEpKalQj95w5CYfCw6OQZs2uzB9+gWo1QIDBtTA5cvfsrAhIgLez3nRNZLm9gUHb8jlckyaNAlTpkxBXFwcunbtCj09PSxYsCBN35UrVyI2Nha9evUCAPTu3RsxMTFYvnx5uusuLBOKWdzkU8eOPYa9/SacPPkURkZ62LSpHdata8vChoioAOjevTt0dHSwbNkylClTBvPmzcPixYsxefJkBAYGIigoCAsXLsS4cePw888/w8nJCQDg5OSkaRs3bhwuXryIJ0+e4MSJE+jevTs2btwo8SvLHdwtlQ9FRMSje/f9iIxMQM2alvD1dUWVKkWljkVERNlEV1cXw4cPx7x58zBkyBD89NNPsLOzw2+//YYlS5ZApVKhevXqWLFiBTw8PFItO3fuXDg6OmLZsmVYuXIl1Go1ypcvj27dusHd3V2iV5S7ZKKQzT6KioqCmZkZIiMjYWpqmn0rDvsHONoAMLYDOgVl33ozsGNHIE6deorFi1vAwICjNURE8fHxePToEcqVKwd9fX2p41AWfOpnqM3nN0du8onDhx9CX18XLVqUAQD07FkFPXtWkTgVERFR3sM5N3lcUpIK48efQfv2e9Cr1wGEhsZKHYmIiChP48hNHvb0aRR69jyAixdfAgC6dasEMzOlxKmIiIjyNhY3edS+fQ/Qv78f3r6Nh5mZEuvWuaBr10pSxyIiIsrzWNzkMSqVGmPHnsGiRVcBAPXqlcCOHR1hZ2cubTAiIqJ8gnNu8hi5XIZXr95f6O2nnxxx7lwvFjZERERa4MhNHpGcrIaurhwymQwrVrRGnz5V0a6dndSxiIiI8h2O3EgsISEZI0acQNeuf2kueGZiomBhQ0RElEUcuZHQgwdv4eZ2AP7+oQCAc+deoEmT0hKnIiIiyt84ciMRH59AODhshr9/KIoWNcCBA1+zsCEiojxr+vTpsLe3lzpGprC4yWVxcUn44Ydj6NnzAKKjE9G4cSkEBPRDhw7lpY5GREQSCgkJwciRI1GhQgXo6+vDysoKjRo1wooVK/Du3Tup42HMmDE4ceKE1DEyhbulclnPngewb18QZDJg4kQnzJjRCLq6rDGJiAqzhw8folGjRjA3N8fs2bNRs2ZNKJVK3LhxA6tXr0apUqXQqVMnSTMaGxvD2NhY0gyZxU/VXDZp0lcoVcoYfn7dMGtWExY2RESEoUOHQldXF//++y969OiBqlWrws7ODp07d8bBgwfh6uqKx48fQyaTISAgQLNcREQEZDIZTp8+rWm7efMm2rVrB2NjY1hZWaFv374ICwvTPL5r1y7UrFkTBgYGKFq0KJydnREb+/7SPqdPn0b9+vVhZGQEc3NzNGrUCE+ePAGQdrdU//790aVLF/z222+wtrZG0aJFMWzYMCQlJWn6BAcHo0OHDjAwMEC5cuWwbds22NraYvHixTmyHVNw5CaHvXuXhCtXQtCsmQ0AwMnJGkFB30Gp5KYnIsoNsbGJGT6moyOHvr5upvrK5TIYGOh9tq+RkUKrfG/evMHRo0cxe/ZsGBkZpdtHJpNlal0RERFo2bIlvvvuOyxatAhxcXEYP348evTogZMnTyI4OBi9evXCvHnz8PXXXyM6Ohpnz56FEALJycno0qULBg0ahO3btyMxMRGXL1/+5HOfOnUK1tbWOHXqFB48eAA3NzfY29tj0KBBAIB+/fohLCwMp0+fhp6eHkaPHo1Xr15ptX2ygp+wOej27TD06LEfQUGRuHSpD2rVKgYALGyIiHKRsfHvGT7Wvn05HDzYVXO/ePHlePcuOd2+zZqVxunTPTX3bW3XICwsLk0/IcZole/BgwcQQqBy5cqp2i0tLREfHw8AGDZsGIYMGfLZdS1duhR16tTB7NmzNW3r16+HjY0N7t27h5iYGCQnJ+Obb75B2bJlAQA1a9YEAISHhyMyMhIdO3ZE+fLv54FWrVr1k89nYWGBpUuXQkdHB1WqVEGHDh1w4sQJDBo0CIGBgTh+/DiuXLmCunXrAgDWrl2LihUrZnLLZF2e2CeybNky2NraQl9fH05OTrh8+fIn++/cuRNVqlSBvr4+atasiUOHDuVS0swRQmDDhhuoW3cLbt16A3NzJaKiEqSORURE+cjly5cREBCA6tWrIyEhc58h169fx6lTpzTzY4yNjVGlShUAQFBQEGrXro1WrVqhZs2a6N69O9asWYO3b98CAIoUKYL+/fvDxcUFrq6uWLJkCYKDgz/5fNWrV4eOjo7mvrW1tWZk5u7du9DV1YWDg4Pm8QoVKsDCwkKr7ZAVkg8h+Pj4YPTo0Vi5ciWcnJywePFiuLi44O7duyhevHia/hcuXECvXr3g5eWFjh07Ytu2bejSpQv8/f1Ro0YNCV5BajFxuhjqfhibN98GALRuXRabN7eHlVX6Q41ERJSzYmJ+zPAxHZ3U3/FfvRqaYV+5PPXumcePB31ZsP9XoUIFyGQy3L17N1W7nd37k7kaGBj8//O/z5pywlcAqea3AEBMTAxcXV0xd+7cNM9jbW0NHR0dHDt2DBcuXMDRo0fxxx9/YPLkybh06RLKlSuHDRs24Mcff4Sfnx98fHwwZcoUHDt2DF999VW62fX09FLdl8lkUKvVWm6B7Cf5yM3ChQsxaNAgeHh4oFq1ali5ciUMDQ2xfv36dPsvWbIEbdu2xdixY1G1alX8+uuvcHBwwNKlS3M5eVr/PbVGvTHdsHnzbcjlMsyc2Rh+ft1Y2BARScjISJHh7cP5Np/r++F8m0/11VbRokXRunVrLF26VDOxNz3Fir2f2vDhaMqHk4sBwMHBAbdu3YKtrS0qVKiQ6pYyn0cmk6FRo0aYMWMGrl27BoVCgb1792rWUadOHUycOBEXLlxAjRo1sG3bNq1fEwBUrlwZycnJuHbtmqbtwYMHmpGinCRpcZOYmIirV6/C2dlZ0yaXy+Hs7IyLFy+mu8zFixdT9QcAFxeXDPsnJCQgKioq1S2n/HW1OgKfW6BkSWOcOtUDkyd/labSJyIi+tjy5cuRnJyMunXrwsfHB3fu3MHdu3exZcsWBAYGQkdHBwYGBvjqq68wZ84c3LlzB2fOnMGUKVNSrWfYsGEIDw9Hr169cOXKFQQFBeHIkSPw8PCASqXCpUuXMHv2bPz77794+vQp9uzZg9evX6Nq1ap49OgRJk6ciIsXL+LJkyc4evQo7t+//9l5NxmpUqUKnJ2dMXjwYFy+fBnXrl3D4MGDYWBgkOkJ0lkl6W6psLAwqFQqWFlZpWq3srJCYGBgusuEhISk2z8kJCTd/l5eXpgxY0b2BP4kGSZ9cx6JOlb48Y/VKFbMMBeek4iICoLy5cvj2rVrmD17NiZOnIjnz59DqVSiWrVqGDNmDIYOfb+7bP369Rg4cCAcHR1RuXJlzJs3D23atNGsp2TJkjh//jzGjx+PNm3aICEhAWXLlkXbtm0hl8thamqKv//+G4sXL0ZUVBTKli2LBQsWoF27dggNDUVgYCA2btyIN2/ewNraGsOGDcP333+f5de1adMmDBw4EE2bNkWJEiXg5eWFW7duQV9f/4u32afIxIc773LZy5cvUapUKVy4cAENGjTQtI8bNw5nzpzBpUuX0iyjUCiwceNG9OrVS9O2fPlyzJgxA6GhoWn6JyQkpJqIFRUVBRsbG0RGRsLU1DSbXxEREUklPj4ejx49Qrly5XL8w5Oy5vnz57CxscHx48fRqlWrNI9/6mcYFRUFMzOzTH1+SzpyY2lpCR0dnTRFSWhoKEqUKJHuMiVKlNCqv1KphFKpzJ7ARERElGknT55ETEwMatasieDgYIwbNw62trZo2rRpjj6vpHNuFAoFHB0dU12rQq1W48SJE6lGcj7UoEGDNNe2OHbsWIb9iYiISBpJSUmYNGkSqlevjq+//hrFihXTnNAvJ0l+KPjo0aPh7u6OunXron79+li8eDFiY2Ph4eEB4P3ZDUuVKgUvLy8AwMiRI9GsWTMsWLAAHTp0wI4dO/Dvv/9i9erVUr4MIiIi+oiLiwtcXFxy/XklL27c3Nzw+vVrTJs2DSEhIbC3t4efn59m0vDTp081x/YDQMOGDbFt2zZMmTIFkyZNQsWKFfHnn3/miXPcEBERkfQknVAsBW0mJBERUf7BCcX5X3ZNKJb8JH5ERETZqZB9Zy9Qsutnx+KGiIgKhJRrHCUmZnxlb8rbUn52H16vKiskn3NDRESUHXR1dWFoaIjXr19DT08v1XxNyvvUajVev34NQ0ND6Op+WXnC4oaIiAoEmUwGa2trPHr0CE+ePJE6DmWBXC5HmTJlvvjyDCxuiIiowFAoFKhYsSJ3TeVTCoUiW0bcWNwQEVGBIpfLebRUIccdkkRERFSgsLghIiKiAoXFDRERERUohW7OTcoJgqKioiROQkRERJmV8rmdmRP9FbriJjo6GgBgY2MjcRIiIiLSVnR0NMzMzD7Zp9BdW0qtVuPly5cwMTH54uPoPxYVFQUbGxs8e/aM163KQdzOuYPbOXdwO+cebuvckVPbWQiB6OholCxZ8rOHixe6kRu5XI7SpUvn6HOYmpryFycXcDvnDm7n3MHtnHu4rXNHTmznz43YpOCEYiIiIipQWNwQERFRgcLiJhsplUp4enpCqVRKHaVA43bOHdzOuYPbOfdwW+eOvLCdC92EYiIiIirYOHJDREREBQqLGyIiIipQWNwQERFRgcLihoiIiAoUFjdaWrZsGWxtbaGvrw8nJydcvnz5k/137tyJKlWqQF9fHzVr1sShQ4dyKWn+ps12XrNmDZo0aQILCwtYWFjA2dn5sz8Xek/b93OKHTt2QCaToUuXLjkbsIDQdjtHRERg2LBhsLa2hlKpRKVKlfi3IxO03c6LFy9G5cqVYWBgABsbG4waNQrx8fG5lDZ/+vvvv+Hq6oqSJUtCJpPhzz///Owyp0+fhoODA5RKJSpUqABvb+8czwlBmbZjxw6hUCjE+vXrxa1bt8SgQYOEubm5CA0NTbf/+fPnhY6Ojpg3b564ffu2mDJlitDT0xM3btzI5eT5i7bbuXfv3mLZsmXi2rVr4s6dO6J///7CzMxMPH/+PJeT5y/abucUjx49EqVKlRJNmjQRnTt3zp2w+Zi22zkhIUHUrVtXtG/fXpw7d048evRInD59WgQEBORy8vxF2+28detWoVQqxdatW8WjR4/EkSNHhLW1tRg1alQuJ89fDh06JCZPniz27NkjAIi9e/d+sv/Dhw+FoaGhGD16tLh9+7b4448/hI6OjvDz88vRnCxutFC/fn0xbNgwzX2VSiVKliwpvLy80u3fo0cP0aFDh1RtTk5O4vvvv8/RnPmdttv5Y8nJycLExERs3LgxpyIWCFnZzsnJyaJhw4Zi7dq1wt3dncVNJmi7nVesWCHs7OxEYmJibkUsELTdzsOGDRMtW7ZM1TZ69GjRqFGjHM1ZkGSmuBk3bpyoXr16qjY3Nzfh4uKSg8mE4G6pTEpMTMTVq1fh7OysaZPL5XB2dsbFixfTXebixYup+gOAi4tLhv0pa9v5Y+/evUNSUhKKFCmSUzHzvaxu519++QXFixfHwIEDcyNmvpeV7bxv3z40aNAAw4YNg5WVFWrUqIHZs2dDpVLlVux8JyvbuWHDhrh69apm19XDhw9x6NAhtG/fPlcyFxZSfQ4WugtnZlVYWBhUKhWsrKxStVtZWSEwMDDdZUJCQtLtHxISkmM587usbOePjR8/HiVLlkzzC0X/k5XtfO7cOaxbtw4BAQG5kLBgyMp2fvjwIU6ePIk+ffrg0KFDePDgAYYOHYqkpCR4enrmRux8JyvbuXfv3ggLC0Pjxo0hhEBycjJ++OEHTJo0KTciFxoZfQ5GRUUhLi4OBgYGOfK8HLmhAmXOnDnYsWMH9u7dC319fanjFBjR0dHo27cv1qxZA0tLS6njFGhqtRrFixfH6tWr4ejoCDc3N0yePBkrV66UOlqBcvr0acyePRvLly+Hv78/9uzZg4MHD+LXX3+VOhplA47cZJKlpSV0dHQQGhqaqj00NBQlSpRId5kSJUpo1Z+ytp1T/Pbbb5gzZw6OHz+OWrVq5WTMfE/b7RwUFITHjx/D1dVV06ZWqwEAurq6uHv3LsqXL5+zofOhrLyfra2toaenBx0dHU1b1apVERISgsTERCgUihzNnB9lZTtPnToVffv2xXfffQcAqFmzJmJjYzF48GBMnjwZcjm/+2eHjD4HTU1Nc2zUBuDITaYpFAo4OjrixIkTmja1Wo0TJ06gQYMG6S7ToEGDVP0B4NixYxn2p6xtZwCYN28efv31V/j5+aFu3bq5ETVf03Y7V6lSBTdu3EBAQIDm1qlTJ7Ro0QIBAQGwsbHJzfj5Rlbez40aNcKDBw80xSMA3Lt3D9bW1ixsMpCV7fzu3bs0BUxKQSl4ycVsI9nnYI5OVy5gduzYIZRKpfD29ha3b98WgwcPFubm5iIkJEQIIUTfvn3FhAkTNP3Pnz8vdHV1xW+//Sbu3LkjPD09eSh4Jmi7nefMmSMUCoXYtWuXCA4O1tyio6Olegn5grbb+WM8WipztN3OT58+FSYmJmL48OHi7t274sCBA6J48eJi5syZUr2EfEHb7ezp6SlMTEzE9u3bxcOHD8XRo0dF+fLlRY8ePaR6CflCdHS0uHbtmrh27ZoAIBYuXCiuXbsmnjx5IoQQYsKECaJv376a/imHgo8dO1bcuXNHLFu2jIeC50V//PGHKFOmjFAoFKJ+/frin3/+0TzWrFkz4e7unqq/r6+vqFSpklAoFKJ69eri4MGDuZw4f9JmO5ctW1YASHPz9PTM/eD5jLbv5w+xuMk8bbfzhQsXhJOTk1AqlcLOzk7MmjVLJCcn53Lq/Eeb7ZyUlCSmT58uypcvL/T19YWNjY0YOnSoePv2be4Hz0dOnTqV7t/blG3r7u4umjVrlmYZe3t7oVAohJ2dndiwYUOO55QJwfE3IiIiKjg454aIiIgKFBY3REREVKCwuCEiIqIChcUNERERFSgsboiIiKhAYXFDREREBQqLGyIiIipQWNwQUSre3t4wNzeXOkaWyWQy/Pnnn5/s079/f3Tp0iVX8hBR7mNxQ1QA9e/fHzKZLM3twYMHUkeDt7e3Jo9cLkfp0qXh4eGBV69eZcv6g4OD0a5dOwDA48ePIZPJEBAQkKrPkiVL4O3tnS3Pl5Hp06drXqeOjg5sbGwwePBghIeHa7UeFmJE2uNVwYkKqLZt22LDhg2p2ooVKyZRmtRMTU1x9+5dqNVqXL9+HR4eHnj58iWOHDnyxev+3NXjAcDMzOyLnyczqlevjuPHj0OlUuHOnTsYMGAAIiMj4ePjkyvPT1RYceSGqIBSKpUoUaJEqpuOjg4WLlyImjVrwsjICDY2Nhg6dChiYmIyXM/169fRokULmJiYwNTUFI6Ojvj33381j587dw5NmjSBgYEBbGxs8OOPPyI2NvaT2WQyGUqUKIGSJUuiXbt2+PHHH3H8+HHExcVBrVbjl19+QenSpaFUKmFvbw8/Pz/NsomJiRg+fDisra2hr6+PsmXLwsvLK9W6U3ZLlStXDgBQp04dyGQyNG/eHEDq0ZDVq1ejZMmSqa7CDQCdO3fGgAEDNPf/+usvODg4QF9fH3Z2dpgxYwaSk5M/+Tp1dXVRokQJlCpVCs7OzujevTuOHTumeVylUmHgwIEoV64cDAwMULlyZSxZskTz+PTp07Fx40b89ddfmlGg06dPAwCePXuGHj16wNzcHEWKFEHnzp3x+PHjT+YhKixY3BAVMnK5HL///jtu3bqFjRs34uTJkxg3blyG/fv06YPSpUvjypUruHr1KiZMmAA9PT0AQFBQENq2bYuuXbviv//+g4+PD86dO4fhw4drlcnAwABqtRrJyclYsmQJFixYgN9++w3//fcfXFxc0KlTJ9y/fx8A8Pvvv2Pfvn3w9fXF3bt3sXXrVtja2qa73suXLwMAjh8/juDgYOzZsydNn+7du+PNmzc4deqUpi08PBx+fn7o06cPAODs2bPo168fRo4cidu3b2PVqlXw9vbGrFmzMv0aHz9+jCNHjkChUGja1Go1SpcujZ07d+L27duYNm0aJk2aBF9fXwDAmDFj0KNHD7Rt2xbBwcEIDg5Gw4YNkZSUBBcXF5iYmODs2bM4f/48jI2N0bZtWyQmJmY6E1GBleOX5iSiXOfu7i50dHSEkZGR5tatW7d0++7cuVMULVpUc3/Dhg3CzMxMc9/ExER4e3unu+zAgQPF4MGDU7WdPXtWyOVyERcXl+4yH6//3r17olKlSqJu3bpCCCFKliwpZs2alWqZevXqiaFDhwohhBgxYoRo2bKlUKvV6a4fgNi7d68QQohHjx4JAOLatWup+nx8RfPOnTuLAQMGaO6vWrVKlCxZUqhUKiGEEK1atRKzZ89OtY7NmzcLa2vrdDMIIYSnp6eQy+XCyMhI6Ovra66evHDhwgyXEUKIYcOGia5du2aYNeW5K1eunGobJCQkCAMDA3HkyJFPrp+oMOCcG6ICqkWLFlixYoXmvpGREYD3oxheXl4IDAxEVFQUkpOTER8fj3fv3sHQ0DDNekaPHo3vvvsOmzdv1uxaKV++PID3u6z+++8/bN26VdNfCAG1Wo1Hjx6hatWq6WaLjIyEsbEx1Go14uPj0bhxY6xduxZRUVF4+fIlGjVqlKp/o0aNcP36dQDvdym1bt0alStXRtu2bdGxY0e0adPmi7ZVnz59MGjQICxfvhxKpRJbt25Fz549IZfLNa/z/PnzqUZqVCrVJ7cbAFSuXBn79u1DfHw8tmzZgoCAAIwYMSJVn2XLlmH9+vV4+vQp4uLikJiYCHt7+0/mvX79Oh48eAATE5NU7fHx8QgKCsrCFiAqWFjcEBVQRkZGqFChQqq2x48fo2PHjhgyZAhmzZqFIkWK4Ny5cxg4cCASExPT/ZCePn06evfujYMHD+Lw4cPw9PTEjh078PXXXyMmJgbff/89fvzxxzTLlSlTJsNsJiYm8Pf3h1wuh7W1NQwMDAAAUVFRn31dDg4OePToEQ4fPozjx4+jR48ecHZ2xq5duz67bEZcXV0hhMDBgwdRr149nD17FosWLdI8HhMTgxkzZuCbb75Js6y+vn6G61UoFJqfwZw5c9ChQwfMmDEDv/76KwBgx44dGDNmDBYsWIAGDRrAxMQE8+fPx6VLlz6ZNyYmBo6OjqmKyhR5ZdI4kZRY3BAVIlevXoVarcaCBQs0oxIp8zs+pVKlSqhUqRJGjRqFXr16YcOGDfj666/h4OCA27dvpymiPkcul6e7jKmpKUqWLInz58+jWbNmmvbz58+jfv36qfq5ubnBzc0N3bp1Q9u2bREeHo4iRYqkWl/K/BaVSvXJPPr6+vjmm2+wdetWPHjwAJUrV4aDg4PmcQcHB9y9e1fr1/mxKVOmoGXLlhgyZIjmdTZs2BBDhw7V9Pl45EWhUKTJ7+DgAB8fHxQvXhympqZflImoIOKEYqJCpEKFCkhKSsIff/yBhw8fYvPmzVi5cmWG/ePi4jB8+HCcPn0aT548wfnz53HlyhXN7qbx48fjwoULGD58OAICAnD//n389ddfWk8o/tDYsWMxd+5c+Pj44O7du5gwYQICAgIwcuRIAMDChQuxfft2BAYG4t69e9i5cydKlCiR7okHixcvDgMDA/j5+SE0NBSRkZEZPm+fPn1w8OBBrF+/XjOROMW0adOwadMmzJgxA7du3cKdO3ewY8cOTJkyRavX1qBBA9SqVQuzZ88GAFSsWBH//vsvjhw5gnv37mHq1Km4cuVKqmVsbW3x33//4e7duwgLC0NSUhL69OkDS0tLdO7cGWfPnsWjR49w+vRp/Pjjj3j+/LlWmYgKJKkn/RBR9ktvEmqKhQsXCmtra2FgYCBcXFzEpk2bBADx9u1bIUTqCb8JCQmiZ8+ewsbGRigUClGyZEkxfPjwVJOFL1++LFq3bi2MjY2FkZGRqFWrVpoJwR/6eELxx1QqlZg+fbooVaqU0NPTE7Vr1xaHDx/WPL569Wphb28vjIyMhKmpqWjVqpXw9/fXPI4PJhQLIcSaNWuEjY2NkMvlolmzZhluH5VKJaytrQUAERQUlCaXn5+faNiwoTAwMBCmpqaifv36YvXq1Rm+Dk9PT1G7du007du3bxdKpVI8ffpUxMfHi/79+wszMzNhbm4uhgwZIiZMmJBquVevXmm2LwBx6tQpIYQQwcHBol+/fsLS0lIolUphZ2cnBg0aJCIjIzPMRFRYyIQQQtryioiIiCj7cLcUERERFSgsboiIiKhAYXFDREREBQqLGyIiIipQWNwQERFRgcLihoiIiAoUFjdERERUoLC4ISIiogKFxQ0REREVKCxuiIiIqEBhcUNEREQFCosbIiIiKlD+D6y9FyBCzd8DAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_roc_curve(fpr, tpr):\n",
    "    \"\"\"\n",
    "    Plots a ROC curve given the false positve rate (fpr) and \n",
    "    true postive rate (tpr) of a classifier.\n",
    "    \"\"\"\n",
    "    # Plot ROC curve\n",
    "    plt.plot(fpr, tpr, color='orange', label='ROC')\n",
    "    # Plot line with no predictive power (baseline)\n",
    "    plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--', label='Guessing')\n",
    "    # Customize the plot\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC) Curve')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "plot_roc_curve(fpr, tpr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the plot for the first time, it might seem a bit confusing.\n",
    "\n",
    "The main thing to take away here is our model is doing far better than guessing.\n",
    "\n",
    "A metric you can use to quantify the ROC curve in a single number is AUC (Area Under Curve). \n",
    "\n",
    "Scikit-Learn implements a function to caculate this called [`sklearn.metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score). \n",
    "\n",
    "The maximum ROC AUC score you can achieve is 1.0 and generally, the closer to 1.0, the better the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.9304956896551724)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "roc_auc_score_value = roc_auc_score(y_test, y_probs)\n",
    "roc_auc_score_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll let you in a secret... \n",
    "\n",
    "Although it was good practice, we didn't actually need to create our own `plot_roc_curve` function.\n",
    "\n",
    "Scikit-Learn allows us to plot a ROC curve directly from our estimator/model by using the class method [`sklearn.metrics.RocCurveDisplay.from_estimator`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.html#sklearn.metrics.RocCurveDisplay.from_estimator) and passing it our `estimator`, `X_test` and `y_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcAAAAGyCAYAAABzzxS5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVpklEQVR4nO3deVxU1f8/8NewzLAvhqyiuCupgJoGbqkoZplLJe5opmluSZa7uOSWS1ia5IJLXw21j5afVFxwSdHMDTcUU1DQQCMUBIQB5vz+8Mf9OLI4gzMgzOv5eNzHgzn33Hvfcxjmzbn33HtkQggBIiIiA2NU0QEQERFVBCZAIiIySEyARERkkJgAiYjIIDEBEhGRQWICJCIig8QESEREBokJkIiIDBITIBERGSSTig6gvKlUKvz999+wtraGTCar6HCIiEhLQgg8fvwYrq6uMDJ6iX6cqEDHjh0T7777rnBxcREAxK5du164zZEjR4SPj4+Qy+Wibt26YsOGDVodMykpSQDgwoULFy6VfElKSipb8vn/KrQHmJWVBS8vL3z00Ufo06fPC+snJCTgnXfewahRo7BlyxZERUXh448/houLCwICAjQ6prW1NQAgKSkJNjY2LxU/ERGVv4yMDLi7u0vf52UlE+LVeBi2TCbDrl270KtXrxLrTJ48GXv27MGVK1eksn79+uHRo0eIjIzU6DgZGRmwtbVFeno6EyARFeuJsgCnE/5FgeqV+Hqk/69ZDTtUt1bo7Hu8Ul0DPHXqFPz9/dXKAgIC8Nlnn5W4TW5uLnJzc6XXGRkZ+gqPiKqI6bsuY+eFexUdBj1nfVBLdG7spLP9VaoEmJKSAicn9Tfv5OSEjIwMPHnyBObm5kW2WbhwIebMmVNeIRJRFfB3+hMAQM1qFrC3MK3gaKiQjblufxeVKgGWxdSpUxEcHCy9Ljx3TET0Il92a4h3m7lWdBikJ5UqATo7O+P+/ftqZffv34eNjU2xvT8AUCgUUCgU5REeERFVIpXqRnhfX19ERUWplR08eBC+vr4VFBEREVVWFZoAMzMzERMTg5iYGABPb3OIiYlBYmIigKenL4cMGSLVHzVqFOLj4/Hll1/i+vXr+P7777F9+3ZMnDixIsInIqJKrEIT4NmzZ+Hj4wMfHx8AQHBwMHx8fDBr1iwAQHJyspQMAaB27drYs2cPDh48CC8vLyxbtgzr1q3T+B5AIiKiQhV6DfCtt95Cabchbty4sdhtLly4oMeoiIjIEFSqa4BERES6wgRIREQGiQmQiIgMUqW6D5B0Z+f5u5j161Uo81UVHQrRK0dZwL8LQ8AEaKCirj1AZm5+RYdB9MoyMzVCI+eXm22AXm1MgAbu8y4N8EHLGhUdBtErx8bMFJYKfkVWZfztGjhbC1O42Bb/GDkioqqMg2CIiMggMQESEZFBYgIkIiKDxARIREQGiQmQiIgMEhMgEREZJCZAIiIySEyARERkkJgAiYjIIDEBEhGRQWICJCIig8QESEREBokJkIiIDBITIBERGSQmQCIiMkhMgEREZJCYAImIyCAxARIRkUFiAiQiIoPEBEhERAaJCZCIiAwSEyARERkkJkAiIjJITIBERGSQmACJiMggMQESEZFBYgIkIiKDxARIREQGiQmQiIgMEhMgEREZJJOKDoBeXm5+AXqujMb1lMcVHQoRUaXBHmAVkJSWXabkpzAxQhM3Wz1ERET06mMPsAqxNTfF4c87aFzfQm4Cc7mxHiMiInp1MQFWIUYy4DUrRUWHQURUKfAUKBERGSQmQCIiMkhMgEREZJCYAImIyCAxARIRkUFiAiQiIoPEBEhERAaJCZCIiAwSEyARERkkJkAiIjJITIBERGSQyvQs0MTERNy5cwfZ2dmoXr06Xn/9dSgUfAYlERFVHhonwNu3b2P16tWIiIjA3bt3IYSQ1snlcrRr1w4jR47E+++/DyMjdiyJiOjVplGmGj9+PLy8vJCQkICvvvoKsbGxSE9Ph1KpREpKCvbu3Yu2bdti1qxZaNasGc6cOaPvuImIiF6KRj1AS0tLxMfH47XXXiuyztHREZ06dUKnTp0QEhKCyMhIJCUl4Y033tB5sERERLqiUQJcuHChxjvs1q1bmYMhIiIqL7xYR0REBklnCfDatWuoU6eOrnZHRESkVzpLgEqlEnfu3NHV7oiIiPRK49sggoODS13/zz//vHQwRERE5UXjHuCKFStw7NgxXLhwodjl+vXrZQpg1apV8PDwgJmZGVq3bo0///yz1PqhoaFo2LAhzM3N4e7ujokTJyInJ6dMxyYiIsOlcQ+wXr16mDhxIgYNGlTs+piYGLRo0UKrg2/btg3BwcEICwtD69atERoaioCAAMTFxcHR0bFI/a1bt2LKlCkIDw+Hn58fbty4gaFDh0Imk2H58uVaHZuIiAybxj3Ali1b4ty5cyWul8lkak+H0cTy5csxYsQIDBs2DJ6enggLC4OFhQXCw8OLrX/y5Em0adMGAwYMgIeHB7p27Yr+/fu/sNdIRET0PI0T4LJly/DZZ5+VuN7LywsqlUrjAyuVSpw7dw7+/v7/C8bICP7+/jh16lSx2/j5+eHcuXNSwouPj8fevXvRvXv3Eo+Tm5uLjIwMtYWIiEjjU6DOzs46PXBqaioKCgrg5OSkVu7k5FTi9cQBAwYgNTUVbdu2hRAC+fn5GDVqFKZNm1bicRYuXIg5c+boNHYiIqr8KtWN8EePHsWCBQvw/fff4/z589i5cyf27NmDefPmlbjN1KlTkZ6eLi1JSUnlGDEREb2qyjQdki44ODjA2NgY9+/fVyu/f/9+ib3NmTNnYvDgwfj4448BAE2bNkVWVhZGjhyJ6dOnFzsLhUKh4FRNRERURIX1AOVyOVq0aIGoqCipTKVSISoqCr6+vsVuk52dXSTJGRsbA4DWA3CIiMiwVVgPEHh6c31QUBBatmyJVq1aITQ0FFlZWRg2bBgAYMiQIXBzc5Mext2jRw8sX74cPj4+aN26NW7evImZM2eiR48eUiIkIiLSRIUmwMDAQPzzzz+YNWsWUlJS4O3tjcjISGlgTGJiolqPb8aMGZDJZJgxYwbu3buH6tWro0ePHpg/f35FvQUiIqqkZKIM5w5///13WFhYoGXLllLZ2bNnkZ2djfbt2+s0QF3LyMiAra0t0tPTYWNjU9Hh6MTNB4/hv/x32FuY4sKsrhUdDhGRXunqe7xMPcC33noLjRo1QmxsrFQ2ePBg3LhxAwUFBWUOhoiIqLyUKQEmJCTA1NRUrSwqKgp5eXk6CYqIiEjfypQAa9WqVaTM1dX1pYMhIiIqL5XqRngiIiJd0agHaG9vD5lMptEO09LSXiogIiKi8qBRAgwNDdVzGEREROVLowQYFBSk7ziIiIjKVZmuAd66dQszZsxA//798eDBAwDAvn37cPXqVZ0GR0REpC9aJ8Bjx46hadOmOH36NHbu3InMzEwAwMWLFxESEqLzAImIiPRB6wQ4ZcoUfPXVVzh48CDkcrlU3qlTJ/zxxx86DY6IiEhftL4P8PLly9i6dWuRckdHR6SmpuokKEMXfTMVm0/dhjJfpVH9rFw+fYeISFtaJ0A7OzskJyejdu3aauUXLlyAm5ubzgIzVD+fu4vJ/7mEApX20zs5WHHeQyIiTWmdAPv164fJkydjx44dkMlkUKlUiI6OxqRJkzBkyBB9xGgQhBD44fd4LNp3HQDQw8sV7es7aLWPN+u8po/QiIiqJK0T4IIFCzBmzBi4u7ujoKAAnp6eKCgowIABAzBjxgx9xFjlqVQC8/dew/oTCQCAke3rYEq3RjAy0uzhA0REpL0yTYcEPJ2r78qVK8jMzISPjw/q16+v69j04lWbDkmZr8IXP1/ErzF/AwCmd2+MEe3rVHBURESvrgqdDgkAatasCXd3dwDQ+DFppC4rNx+j/u8cjv+VChMjGZZ82Ay9fWpUdFhERAahTDfCr1+/Hk2aNIGZmRnMzMzQpEkTrFu3TtexVWn/ZuZiwNo/cPyvVJibGmNdUEsmPyKicqR1D3DWrFlYvnw5xo0bB19fXwDAqVOnMHHiRCQmJmLu3Lk6D7KqSUrLxpDwP5GQmgV7C1OED30DPjXtKzosIiKDovU1wOrVq+Pbb79F//791cp/+uknjBs37pW/F7CirwFeS85AUPifePA4F2525tg8vBXqVrcq9ziIiCqrCrsGmJeXh5YtWxYpb9GiBfLz88sciCH4I/5fjNh8Fo9z8tHQyRqbPmoFZ1uzig6LiMggaX0NcPDgwVi9enWR8jVr1mDgwIE6CaoqirySgiHhf+JxTj7e8LDH9k98mfyIiCqQRj3A4OBg6WeZTIZ169bhwIEDePPNNwEAp0+fRmJiIm+EL8FPfyZi+q7LUAmgi6cTvuvvAzNT44oOi4jIoGmUAC9cuKD2ukWLFgCeTosEAA4ODnBwcOB0SMX4NzMXM3+5ApUA+r3hjq96NYGJcZkG3xIRkQ5plACPHDmi7ziqrEdP8pCvErBWmGBhn6a8Z5KI6BXBrkg5kcn4wAAioldJmZ4Ec/bsWWzfvh2JiYlQKpVq63bu3KmTwIiIiPRJ6x5gREQE/Pz8cO3aNezatQt5eXm4evUqDh8+DFtbW33ESEREpHNaJ8AFCxbgm2++wX//+1/I5XKsWLEC169fR9++fVGzZk19xEhERKRzWifAW7du4Z133gEAyOVyZGVlQSaTYeLEiVizZo3OAyQiItIHrROgvb09Hj9+DABwc3PDlStXAACPHj1Cdna2bqMjIiLSE60HwbRv3x4HDx5E06ZN8eGHH2LChAk4fPgwDh48iM6dO+sjRiIiIp3TOgGuXLkSOTk5AIDp06fD1NQUJ0+exPvvv88Z4YmIqNLQOgFWq1ZN+tnIyAhTpkzRaUCVQbYyH3kFmk2ikZnDB4QTEb2KNEqAGRkZGu+wIqYYKk9bTt+RHm1GRESVl0YJ0M7O7oVPMRFCQCaToaCgQCeBvar+TEgrU/Lr1MhR98EQEVGZ8VmgZTSteyMM9autcX25CZ86R0T0KtEoAXbo0EHfcVQ6xkZGTGpERJUYv8GJiMggMQESEZFBYgIkIiKDxARIREQGqUwJMD8/H4cOHcIPP/wgPRf077//RmZmpk6DIyIi0hetnwRz584ddOvWDYmJicjNzUWXLl1gbW2NxYsXIzc3F2FhYfqIk4iISKe07gFOmDABLVu2xMOHD2Fubi6V9+7dG1FRUToNjoiISF+07gEeP34cJ0+ehFwuVyv38PDAvXv3dBYYERGRPmndA1SpVMU+7uzu3buwtrbWSVBERET6pnUC7Nq1K0JDQ6XXMpkMmZmZCAkJQffu3XUZGxERkd5ofQp02bJlCAgIgKenJ3JycjBgwAD89ddfcHBwwE8//aSPGImIiHRO6wRYo0YNXLx4EREREbh06RIyMzMxfPhwDBw4UG1QDBER0atM6wSYk5MDMzMzDBo0SB/xEBERlQutrwE6OjoiKCgIBw8ehEql0kdMREREeqd1Aty0aROys7PRs2dPuLm54bPPPsPZs2f1ERsREZHeaJ0Ae/fujR07duD+/ftYsGABYmNj8eabb6JBgwaYO3euPmIkIiLSuTI/DNva2hrDhg3DgQMHcOnSJVhaWmLOnDm6jI2IiEhvypwAc3JysH37dvTq1QvNmzdHWloavvjiC13GRkREpDdajwLdv38/tm7dil9++QUmJib44IMPcODAAbRv314f8REREemF1gmwd+/eePfdd7F582Z0794dpqam+oiLiIhIr7ROgPfv3+czP4mIqNLTKAFmZGTAxsYGACCEQEZGRol1C+sRERG9yjRKgPb29khOToajoyPs7Owgk8mK1BFCQCaTFTtTBBER0atGowR4+PBhVKtWDQBw5MgRvQZERERUHjRKgB06dJB+rl27Ntzd3Yv0AoUQSEpK0m10REREeqL1fYC1a9fGP//8U6Q8LS0NtWvX1jqAVatWwcPDA2ZmZmjdujX+/PPPUus/evQIY8aMgYuLCxQKBRo0aIC9e/dqfVwiIjJsWo8CLbzW97zMzEyYmZlpta9t27YhODgYYWFhaN26NUJDQxEQEIC4uDg4OjoWqa9UKtGlSxc4Ojri559/hpubG+7cuQM7Oztt3wYRERk4jRNgcHAwgKczwM+cORMWFhbSuoKCApw+fRre3t5aHXz58uUYMWIEhg0bBgAICwvDnj17EB4ejilTphSpHx4ejrS0NJw8eVK6/9DDw0OrYxIREQFaJMALFy4AeNoDvHz5MuRyubROLpfDy8sLkyZN0vjASqUS586dw9SpU6UyIyMj+Pv749SpU8Vus3v3bvj6+mLMmDH49ddfUb16dQwYMACTJ0+GsbFxsdvk5uYiNzdXel3aLRxERGQ4NE6AhaM/hw0bhhUrVrz0/X6pqakoKCiAk5OTWrmTkxOuX79e7Dbx8fE4fPgwBg4ciL179+LmzZv49NNPkZeXh5CQkGK3WbhwIR/STURERWg9CGbDhg0VdrO7SqWCo6Mj1qxZgxYtWiAwMBDTp09HWFhYidtMnToV6enp0sKRqkREBGjYA+zTpw82btwIGxsb9OnTp9S6O3fu1OjADg4OMDY2xv3799XK79+/D2dn52K3cXFxgampqdrpzsaNGyMlJQVKpVLttGwhhUIBhUKhUUxERGQ4NOoB2traSiM/bW1tS100JZfL0aJFC0RFRUllKpUKUVFR8PX1LXabNm3a4ObNm1CpVFLZjRs34OLiUmzyIyIiKolGPcANGzYU+/PLCg4ORlBQEFq2bIlWrVohNDQUWVlZ0qjQIUOGwM3NDQsXLgQAjB49GitXrsSECRMwbtw4/PXXX1iwYAHGjx+vs5iIiMgwaH0f4JMnTyCEkG6DuHPnDnbt2gVPT0907dpVq30FBgbin3/+waxZs5CSkgJvb29ERkZKA2MSExNhZPS/Tqq7uzv279+PiRMnolmzZnBzc8OECRMwefJkbd8GEREZOJkQQmizQdeuXdGnTx+MGjUKjx49QsOGDSGXy5Gamorly5dj9OjR+opVJzIyMmBra4v09PQyDeaZEHEBv8b8jZnvemJ4W+2ffENERC/nZb/HC2k9CvT8+fNo164dAODnn3+Gs7Mz7ty5g82bN+Pbb78tcyBERETlSesEmJ2dLU2Ie+DAAfTp0wdGRkZ48803cefOHZ0HSEREpA9aJ8B69erhl19+QVJSEvbv3y9d93vw4AEnwyUiokpD6wQ4a9YsTJo0CR4eHmjVqpV0y8KBAwfg4+Oj8wCJiIj0QetRoB988AHatm2L5ORkeHl5SeWdO3dG7969dRocERGRvmidAAHA2dkZzs7OuHv3LgCgRo0aaNWqlU4DIyIi0ietT4GqVCrMnTsXtra2qFWrFmrVqgU7OzvMmzdP7QktRERErzKte4DTp0/H+vXrsWjRIrRp0wYAcOLECcyePRs5OTmYP3++zoMkIiLSNa0T4KZNm7Bu3Tq89957UlnhU1k+/fRTJkAiIqoUtD4FmpaWhkaNGhUpb9SoEdLS0nQSFBERkb5pnQC9vLywcuXKIuUrV65UGxVKRET0KtP6FOjXX3+Nd955B4cOHZLuATx16hSSkpKwd+9enQdIRESkD1r3ADt06IAbN26gT58+ePToER49eoQ+ffogLi5OekYoERHRq06rHuDt27dx8OBBKJVK9OvXD02aNNFXXERERHqlcQI8cuQI3n33XTx58uTphiYmCA8Px6BBg/QWHBERkb5ofAp05syZ6NKlC+7du4d///0XI0aMwJdffqnP2IiIiPRG4wR45coVLFiwAC4uLrC3t8eSJUvw4MED/Pvvv/qMj4iISC80ToAZGRlwcHCQXltYWMDc3Bzp6el6CYyIiEiftBoEs3//ftja2kqvVSoVoqKicOXKFans2SfEEBERvaq0SoBBQUFFyj755BPpZ5lMhoKCgpePioiISM80ToCc6YGIiKoSrW+EJyIiqgo0SoB//PGHxjvMzs7G1atXyxwQERFRedAoAQ4ePBgBAQHYsWMHsrKyiq0TGxuLadOmoW7dujh37pxOgyQiItI1ja4BxsbGYvXq1ZgxYwYGDBiABg0awNXVFWZmZnj48CGuX7+OzMxM9O7dGwcOHEDTpk31HTcREdFL0SgBmpqaYvz48Rg/fjzOnj2LEydO4M6dO3jy5Am8vLwwceJEdOzYEdWqVdN3vERERDqh9XRILVu2RMuWLfURCxERUbnhKFAiIjJITIBERGSQmACJiMggMQESEZFBeqkEmJOTo6s4iIiIypXWCVClUmHevHlwc3ODlZUV4uPjATydMHf9+vU6D5CIiEgftE6AX331FTZu3Iivv/4acrlcKm/SpAnWrVun0+CIiIj0ResEuHnzZqxZswYDBw6EsbGxVO7l5YXr16/rNDgiIiJ90ToB3rt3D/Xq1StSrlKpkJeXp5OgiIiI9E3rBOjp6Ynjx48XKf/555/h4+Ojk6CIiIj0TetHoc2aNQtBQUG4d+8eVCoVdu7cibi4OGzevBm//fabPmIkIiLSOa17gD179sR///tfHDp0CJaWlpg1axauXbuG//73v+jSpYs+YiQiItI5rXuAANCuXTscPHhQ17EQERGVG617gHXq1MG///5bpPzRo0eoU6eOToIiIiLSN60T4O3bt1FQUFCkPDc3F/fu3dNJUERERPqm8SnQ3bt3Sz/v378ftra20uuCggJERUXBw8NDp8ERERHpi8YJsFevXgAAmUyGoKAgtXWmpqbw8PDAsmXLdBocERGRvmicAFUqFQCgdu3aOHPmDBwcHPQWFBERkb5pPQo0ISFBH3EQERGVqzLdBpGVlYVjx44hMTERSqVSbd348eN1EhgREZE+aZ0AL1y4gO7duyM7OxtZWVmoVq0aUlNTYWFhAUdHRyZAIiKqFLS+DWLixIno0aMHHj58CHNzc/zxxx+4c+cOWrRogaVLl+ojRiIiIp3TOgHGxMTg888/h5GREYyNjZGbmwt3d3d8/fXXmDZtmj5iJCIi0jmtT4GamprCyOhp3nR0dERiYiIaN24MW1tbJCUl6TxAfTty/QFikzM0rh+X8liP0RARUXnROgH6+PjgzJkzqF+/Pjp06IBZs2YhNTUVP/74I5o0aaKPGPUmNTMXH206AyG039bMVOvOMxERvUK0ToALFizA48dPe0Hz58/HkCFDMHr0aNSvXx/r16/XeYD6lJmTDyEAEyMZ3m9eQ+Pt7CxN8W5TVz1GRkRE+qZ1AmzZsqX0s6OjIyIjI3UaUEUwNzXG4g+aVXQYRERUjnR2Hu/8+fN49913dbU7IiIivdIqAe7fvx+TJk3CtGnTEB8fDwC4fv06evXqhTfeeEN6XBoREdGrTuNToOvXr8eIESNQrVo1PHz4EOvWrcPy5csxbtw4BAYG4sqVK2jcuLE+YyUiItIZjXuAK1aswOLFi5Gamort27cjNTUV33//PS5fvoywsDAmPyIiqlQ0ToC3bt3Chx9+CADo06cPTExMsGTJEtSoofnoSSIioleFxgnwyZMnsLCwAPB0TkCFQgEXFxe9BUZERKRPWt0GsW7dOlhZWQEA8vPzsXHjxiLzAvJh2EREVBnIhNDsOSgeHh6QyWSl70wmk0aHamPVqlVYsmQJUlJS4OXlhe+++w6tWrV64XYRERHo378/evbsiV9++UWjY2VkZMDW1hbp6elIUxrjraVHYa0wweU5AVrHTURE5e/Z73EbG5sy70fjHuDt27fLfJDSbNu2DcHBwQgLC0Pr1q0RGhqKgIAAxMXFwdHRsdR4Jk2ahHbt2uklLiIiqtoq/IGWy5cvx4gRIzBs2DB4enoiLCwMFhYWCA8PL3GbgoICDBw4EHPmzEGdOnXKMVoiIqoqKjQBKpVKnDt3Dv7+/lKZkZER/P39cerUqRK3mzt3LhwdHTF8+PAXHiM3NxcZGRlqCxERUYUmwNTUVBQUFMDJyUmt3MnJCSkpKcVuc+LECaxfvx5r167V6BgLFy6Era2ttLi7u7903EREVPlV+ClQbTx+/BiDBw/G2rVri4w+LcnUqVORnp4uLZVxzkIiItI9rWeD0CUHBwcYGxvj/v37auX379+Hs7Nzkfq3bt3C7du30aNHD6ms8PmjJiYmiIuLQ926ddW2USgUUCgUeoieiIgqszL1AG/duoUZM2agf//+ePDgAQBg3759uHr1qlb7kcvlaNGiBaKioqQylUqFqKgo+Pr6FqnfqFEjXL58GTExMdLy3nvvoWPHjoiJieHpTSIi0pjWCfDYsWNo2rQpTp8+jZ07dyIzMxMAcPHiRYSEhGgdQHBwMNauXYtNmzbh2rVrGD16NLKysjBs2DAAwJAhQzB16lQAgJmZGZo0aaK22NnZwdraGk2aNIFcLtf6+EREZJi0PgU6ZcoUfPXVVwgODoa1tbVU3qlTJ6xcuVLrAAIDA/HPP/9g1qxZSElJgbe3NyIjI6WBMYmJiTAyqlSXKomIqBLQ+EkwhaysrHD58mXUrl0b1tbWuHjxIurUqYPbt2+jUaNGyMnJ0VesOsEnwRARVW66ehKM1l0rOzs7JCcnFym/cOEC3NzcyhwIERFRedI6Afbr1w+TJ09GSkoKZDIZVCoVoqOjMWnSJAwZMkQfMRIREemc1glwwYIFaNSoEdzd3ZGZmQlPT0+0b98efn5+mDFjhj5iJCIi0jmtB8HI5XKsXbsWM2fOxJUrV5CZmQkfHx/Ur19fH/ERERHphdYJ8MSJE2jbti1q1qyJmjVr6iMmIiIivdP6FGinTp1Qu3ZtTJs2DbGxsfqIiYiISO+0ToB///03Pv/8cxw7dgxNmjSBt7c3lixZgrt37+ojPiIiIr3QOgE6ODhg7NixiI6Oxq1bt/Dhhx9i06ZN8PDwQKdOnfQRIxERkc691CNWateujSlTpmDRokVo2rQpjh07pqu4iIiI9KrMCTA6OhqffvopXFxcMGDAADRp0gR79uzRZWxERER6o/Uo0KlTpyIiIgJ///03unTpghUrVqBnz56wsLDQR3xERER6oXUC/P333/HFF1+gb9++Gk9KS0RE9KrROgFGR0frIw4iIqJypVEC3L17N95++22Ymppi9+7dpdZ97733dBIYERGRPmmUAHv16oWUlBQ4OjqiV69eJdaTyWQoKCjQVWxERER6o1ECVKlUxf5MRERUWWl9G8TmzZuRm5tbpFypVGLz5s06CYqIiEjftE6Aw4YNQ3p6epHyx48fY9iwYToJioiISN+0ToBCCMhksiLld+/eha2trU6CIiIi0jeNb4Pw8fGBTCaDTCZD586dYWLyv00LCgqQkJCAbt266SVIIiIiXdM4ARaO/oyJiUFAQACsrKykdXK5HB4eHnj//fd1HiAREZE+aJwAQ0JCAAAeHh4IDAyEmZmZ3oIiIiLSN62fBBMUFKSPOIiIiMqVRgmwWrVquHHjBhwcHGBvb1/sIJhCaWlpOguOiIhIXzRKgN988w2sra2ln0tLgERERJWBRgnw2dOeQ4cO1VcsRERE5Ubr+wDPnz+Py5cvS69//fVX9OrVC9OmTYNSqdRpcERERPqidQL85JNPcOPGDQBAfHw8AgMDYWFhgR07duDLL7/UeYBERET6oHUCvHHjBry9vQEAO3bsQIcOHbB161Zs3LgR//nPf3QdHxERkV6U6VFohTNCHDp0CN27dwcAuLu7IzU1VbfRERER6YnWCbBly5b46quv8OOPP+LYsWN45513AAAJCQlwcnLSeYBERET6oHUCDA0Nxfnz5zF27FhMnz4d9erVAwD8/PPP8PPz03mARERE+qD1k2CaNWumNgq00JIlS2BsbKyToIiIiPRN6wRY6Ny5c7h27RoAwNPTE82bN9dZUERERPqmdQJ88OABAgMDcezYMdjZ2QEAHj16hI4dOyIiIgLVq1fXdYxEREQ6p/U1wHHjxiEzMxNXr15FWloa0tLScOXKFWRkZGD8+PH6iJGIiEjntO4BRkZG4tChQ2jcuLFU5unpiVWrVqFr1646DY6IiEhftO4BqlQqmJqaFik3NTWV7g8kIiJ61WmdADt16oQJEybg77//lsru3buHiRMnonPnzjoNjoiISF+0ToArV65ERkYGPDw8ULduXdStWxe1a9dGRkYGvvvuO33ESEREpHNaXwN0d3fH+fPnERUVJd0G0bhxY/j7++s8OCIiIn3RKgFu27YNu3fvhlKpROfOnTFu3Dh9xUVERKRXGifA1atXY8yYMahfvz7Mzc2xc+dO3Lp1C0uWLNFnfERERHqh8TXAlStXIiQkBHFxcYiJicGmTZvw/fff6zM2IiIivdE4AcbHxyMoKEh6PWDAAOTn5yM5OVkvgREREemTxgkwNzcXlpaW/9vQyAhyuRxPnjzRS2BERET6pNUgmJkzZ8LCwkJ6rVQqMX/+fNja2kply5cv1110REREeqJxAmzfvj3i4uLUyvz8/BAfHy+9lslkuouMiIhIjzROgEePHtVjGEREROVL6yfBEBERVQVMgEREZJCYAImIyCAxARIRkUFiAiQiIoNUpgR4/PhxDBo0CL6+vrh37x4A4Mcff8SJEyd0GhwREZG+aJ0A//Of/yAgIADm5ua4cOECcnNzAQDp6elYsGCBzgMkIiLSB60T4FdffYWwsDCsXbsWpqamUnmbNm1w/vx5nQZHRESkL1onwLi4OLRv375Iua2tLR49eqSLmIiIiPRO6wTo7OyMmzdvFik/ceIE6tSpo5OgiIiI9E3rBDhixAhMmDABp0+fhkwmw99//40tW7Zg0qRJGD16tD5iJCIi0jmtZoMAgClTpkClUqFz587Izs5G+/btoVAoMGnSJIwbN04fMRIREemc1glQJpNh+vTp+OKLL3Dz5k1kZmbC09MTVlZW+oiPiIhIL8p8I7xcLoenpydatWr10slv1apV8PDwgJmZGVq3bo0///yzxLpr165Fu3btYG9vD3t7e/j7+5dan4iIqDha9wA7duxY6rx/hw8f1mp/27ZtQ3BwMMLCwtC6dWuEhoYiICAAcXFxcHR0LFL/6NGj6N+/P/z8/GBmZobFixeja9euuHr1Ktzc3LR9O0REZKC07gF6e3vDy8tLWjw9PaFUKnH+/Hk0bdpU6wCWL1+OESNGYNiwYfD09ERYWBgsLCwQHh5ebP0tW7bg008/hbe3Nxo1aoR169ZBpVIhKipK62MTEZHh0roH+M033xRbPnv2bGRmZmq1L6VSiXPnzmHq1KlSmZGREfz9/XHq1CmN9pGdnY28vDxUq1at2PW5ubnS02oAICMjQ6sYiYioatLZw7AHDRpUYq+tJKmpqSgoKICTk5NauZOTE1JSUjTax+TJk+Hq6gp/f/9i1y9cuBC2trbS4u7urlWMRERUNeksAZ46dQpmZma62p1GFi1ahIiICOzatavEY0+dOhXp6enSkpSUVK4xEhHRq0nrU6B9+vRRey2EQHJyMs6ePYuZM2dqtS8HBwcYGxvj/v37auX379+Hs7NzqdsuXboUixYtwqFDh9CsWbMS6ykUCigUCq3iIiKiqk/rHuCzpxNtbW1RrVo1vPXWW9i7dy9CQkK02pdcLkeLFi3UBrAUDmjx9fUtcbuvv/4a8+bNQ2RkJFq2bKntWyAiItKuB1hQUIBhw4ahadOmsLe310kAwcHBCAoKQsuWLdGqVSuEhoYiKysLw4YNAwAMGTIEbm5uWLhwIQBg8eLFmDVrFrZu3QoPDw/pWqGVlRVvxiciIo1plQCNjY3RtWtXXLt2TWcJMDAwEP/88w9mzZqFlJQUeHt7IzIyUhoYk5iYCCOj/3VUV69eDaVSiQ8++EBtPyEhIZg9e7ZOYiIioqpP62uATZo0QXx8PGrXrq2zIMaOHYuxY8cWu+7o0aNqr2/fvq2z4xIRkeEq04S4kyZNwm+//Ybk5GRkZGSoLURERJWBxj3AuXPn4vPPP0f37t0BAO+9957aI9GEEJDJZCgoKNB9lERERDqmcQKcM2cORo0ahSNHjugzHiIionKhcQIUQgAAOnTooLdgiIiIyotW1wBLmwWCiIioMtFqFGiDBg1emATT0tJeKiAiIqLyoFUCnDNnDmxtbfUVCxERUbnRKgH269ev2ElqiYiIKhuNrwHy+h8REVUlGifAwlGgREREVYHGp0BVKpU+4yAiIipXOpsQl4iIqDJhAiQiIoPEBEhERAaJCZCIiAwSEyARERkkJkAiIjJITIBERGSQmACJiMggMQESEZFBYgIkIiKDxARIREQGiQmQiIgMEhMgEREZJCZAIiIySEyARERkkJgAiYjIIDEBEhGRQWICJCIig8QESEREBokJkIiIDBITIBERGSQmQCIiMkhMgEREZJCYAImIyCAxARIRkUEyqegASHeEEMjPz0dBQUFFh0JEVGbGxsYwMTGBTCbT63GYAKsIpVKJ5ORkZGdnV3QoREQvzcLCAi4uLpDL5Xo7BhNgFaBSqZCQkABjY2O4urpCLpfr/T8nIiJ9EEJAqVTin3/+QUJCAurXrw8jI/1crWMCrAKUSiVUKhXc3d1hYWFR0eEQEb0Uc3NzmJqa4s6dO1AqlTAzM9PLcTgIpgrR139JRETlrTy+z/iNSUREBokJkIiIDBITIBk0mUyGX375paLDqHTeeustfPbZZ+VyrOd/R9evX8ebb74JMzMzeHt74/bt25DJZIiJidHL8ZVKJerVq4eTJ0/qZf+GaMqUKRg3blxFh8EESBVr6NChkMlkkMlkMDU1Re3atfHll18iJyenokPTqcL3+OzStm3bCo+puOSvVCrx9ddfw8vLCxYWFnBwcECbNm2wYcMG5OXllXucycnJePvtt6XXISEhsLS0RFxcHKKiouDu7o7k5GQ0adJEL8cPCwtD7dq14efnV2TdJ598AmNjY+zYsaPIuqFDh6JXr15Fyo8ePQqZTIZHjx5JZRXV5pcuXUK7du1gZmYGd3d3fP311y/cJioqCn5+frC2toazszMmT56M/Px8aX1cXBw6duwIJycnmJmZoU6dOpgxY4ba+5g0aRI2bdqE+Ph4vbwvTXEUKFW4bt26SX/o586dQ1BQEGQyGRYvXlzRoenUhg0b0K1bN+n1y9zflJeXB1NTU12EpUapVCIgIAAXL17EvHnz0KZNG9jY2OCPP/7A0qVL4ePjA29vb50ftzTOzs5qr2/duoV33nkHtWrVKrGOtpRKZbG/DyEEVq5ciblz5xZZl52djYiICHz55ZcIDw/Hhx9+WOZjV0SbZ2RkoGvXrvD390dYWBguX76Mjz76CHZ2dhg5cmSx21y8eBHdu3fH9OnTsXnzZty7dw+jRo1CQUEBli5dCgAwNTXFkCFD0Lx5c9jZ2eHixYsYMWIEVCoVFixYAABwcHBAQEAAVq9ejSVLluj8vWlMGJj09HQBQKSnp4uEfzJFrcm/iSazIis6rJfy5MkTERsbK548eSKVqVQqkZWbVyGLSqXSOPagoCDRs2dPtbI+ffoIHx8f6XVqaqro16+fcHV1Febm5qJJkyZi69atatt06NBBjBs3TnzxxRfC3t5eODk5iZCQELU6N27cEO3atRMKhUI0btxYHDhwQAAQu3btkupcunRJdOzYUZiZmYlq1aqJESNGiMePHxeJd/78+cLR0VHY2tqKOXPmiLy8PDFp0iRhb28v3NzcRHh4uNqxnz/OswoKCsScOXOEm5ubkMvlwsvLS+zbt09an5CQIACIiIgI0b59e6FQKMSGDRuEEEKsXbtWNGrUSCgUCtGwYUOxatUqabvc3FwxZswY4ezsLBQKhahZs6ZYsGCBEEKIWrVqCQDSUqtWLSGEEIsXLxZGRkbi/PnzReJUKpUiMzNTau8JEyZI6zZv3ixatGghrKyshJOTk+jfv7+4f/++tD4tLU0MGDBAODg4CDMzM1GvXj2pjUqL8/m2ezZmACIkJERqnwsXLkjbXL58WXTr1k1YWloKR0dHMWjQIPHPP/9I6zt06CDGjBkjJkyYIF577TXx1ltvFfu7OXPmjDAyMhIZGRlF1m3cuFG8+eab4tGjR8LCwkIkJiaqrS/usy2EEEeOHBEAxMOHD7Vqc137/vvvhb29vcjNzZXKJk+eLBo2bFjiNlOnThUtW7ZUK9u9e7cwMzMrto0KTZw4UbRt21atbNOmTaJGjRolblPc91qhZ7/HXwZ7gFXUk7wCeM7aXyHHjp0bAAt52T5aV65cwcmTJ9X+u8/JyUGLFi0wefJk2NjYYM+ePRg8eDDq1q2LVq1aSfU2bdqE4OBgnD59GqdOncLQoUPRpk0bdOnSBSqVCn369IGTkxNOnz6N9PT0ItewsrKyEBAQAF9fX5w5cwYPHjzAxx9/jLFjx2Ljxo1SvcOHD6NGjRr4/fffER0djeHDh+PkyZNo3749Tp8+jW3btuGTTz5Bly5dUKNGjRe+5xUrVmDZsmX44Ycf4OPjg/DwcLz33nu4evUq6tevL9WbMmUKli1bBh8fH5iZmWHLli2YNWsWVq5cCR8fH1y4cAEjRoyApaUlgoKC8O2332L37t3Yvn07atasiaSkJCQlJQEAzpw5A0dHR6lXamxsDADYsmUL/P394ePjUyROU1PTEnudeXl5mDdvHho2bIgHDx4gODgYQ4cOxd69ewEAM2fORGxsLPbt2wcHBwfcvHkTT548AYBS43xecnIy/P390a1bN0yaNAlWVlZITU1Vq/Po0SN06tQJH3/8Mb755hs8efIEkydPRt++fXH48GGp3qZNmzB69GhER0eX+Ls5fvw4GjRoAGtr6yLr1q9fj0GDBsHW1hZvv/02Nm7ciJkzZ5a4r5KUtc0TExPh6elZ6r6nTZuGadOmFbvu1KlTaN++vVrPNyAgAIsXL8bDhw9hb29fZJvc3Nwi9+SZm5sjJycH586dw1tvvVVkm5s3byIyMhJ9+vRRK2/VqhXu3r2L27dvw8PDo9T3oS9MgFThfvvtN1hZWSE/Px+5ubkwMjLCypUrpfVubm6YNGmS9HrcuHHYv38/tm/frpYAmzVrhpCQEABA/fr1sXLlSkRFRaFLly44dOgQrl+/jv3798PV1RUAsGDBArVrS1u3bkVOTg42b94MS0tLAMDKlSvRo0cPLF68GE5OTgCAatWq4dtvv4WRkREaNmyIr7/+GtnZ2dIXzdSpU7Fo0SKcOHEC/fr1k/bfv39/KdEAwP/93/+hV69eWLp0KSZPnizVXbx4MY4cOYLQ0FCsWrVKqv/ZZ5+pfYmEhIRg2bJlUlnt2rURGxuLH374AUFBQUhMTET9+vXRtm1byGQytX8qqlevDgCws7NTO334119/Ffsl9iIfffSR9HOdOnXw7bff4o033kBmZiasrKyQmJgIHx8ftGzZEgDUvvBKi/N5zs7OMDExgZWVlRT38wmw8B+CwtNtABAeHg53d3fcuHEDDRo0APD0M/Kia1537tyRPi/P+uuvv/DHH39g586dAIBBgwYhODgYM2bM0PopTGVtc1dX1xcO/KlWrVqJ61JSUlC7dm21ssLPeEpKSrEJMCAgAKGhofjpp5/Qt29fpKSkSKeHk5OT1er6+fnh/PnzyM3NxciRI4ucRi5s1zt37jABkm6Zmxojdm5AhR1bGx07dsTq1auRlZWFb775BiYmJnj//fel9QUFBViwYAG2b9+Oe/fuQalUIjc3t8hTb5o1a6b22sXFBQ8ePAAAXLt2De7u7mpfZr6+vmr1r127Bi8vLyn5AUCbNm2gUqkQFxcnfTm8/vrrajfpOjk5qQ3AMDY2xmuvvSYdu9A333wDf39/tfgyMjLw999/o02bNmp127Rpg4sXL6qVFSYP4Glv9datWxg+fDhGjBghlefn58PW1hbA00EYXbp0QcOGDdGtWze8++676Nq1K0ojhCh1fUnOnTuH2bNn4+LFi3j48CFUKhWA//VSRo8ejffffx/nz59H165d0atXL2lQSVniLM3Fixdx5MgRWFlZFVl369YtKQG2aNHihft68uRJsU8hCQ8PR0BAABwcHAAA3bt3x/Dhw3H48GF07txZq3jL2uYmJiaoV69embYtq65du2LJkiUYNWoUBg8eDIVCgZkzZ+L48eNFblzftm0bHj9+jIsXL+KLL77A0qVL8eWXX0rrzc3NAaBCn1/MBFhFyWSyMp+GLG+WlpbSH3J4eDi8vLywfv16DB8+HACwZMkSrFixAqGhoWjatCksLS3x2WefQalUqu3n+VNFMplM+iLWpeKOo8mxnZ2di3xhZWRkaHzcZxNzZmYmAGDt2rVo3bq1Wr3CXmbz5s2RkJCAffv24dChQ+jbty/8/f3x888/l3iMBg0a4Pr16xrHBPzv1HFAQAC2bNmC6tWrIzExEQEBAdLv6O2338adO3ewd+9eHDx4EJ07d8aYMWOwdOnSMsVZmszMTKnX/jwXFxfp52fbsyQODg64fPmyWllBQQE2bdqElJQUmJiYqJWHh4dLCdDGxgZ37twpss9Hjx7B2NhYOn5Z2hx4+VOgzs7OuH//vlpZ4evSBhUFBwdj4sSJSE5Ohr29PW7fvo2pU6eiTp06avXc3d0BAJ6enigoKMDIkSPx+eefS5/PtLQ0AP87G1ERKsc3JBkMIyMjTJs2DcHBwRgwYADMzc0RHR2Nnj17YtCgQQCePvz7xo0bL/zjf1bjxo2RlJSE5ORk6Uvwjz/+KFJn48aNyMrKkr6coqOjpVOd+mBjYwNXV1dER0ejQ4cOUnl0dLTa6d3nOTk5wdXVFfHx8Rg4cGCp+w8MDERgYCA++OADdOvWDWlpaahWrRpMTU2LTJ01YMAATJs2DRcuXChyTSovLw9KpbJI4rh+/Tr+/fdfLFq0SPrSO3v2bJFYqlevjqCgIAQFBaFdu3ZSr+BFcWqrefPm+M9//gMPDw+1BFUWPj4+WL16NYQQ0qnNvXv34vHjx7hw4YLaKe0rV65g2LBhePToEezs7NCwYUNEREQgNzcXCoVCqnf+/HnUrl1b+qepLG0OvPwpUF9fX0yfPl1tRPHBgwfRsGHDYk9/Pksmk0lnU3766Se4u7ujefPmJdZXqVTIy8uDSqWS2uzKlSswNTXF66+/Xuqx9In3AdIr58MPP4SxsbF0/at+/fo4ePAgTp48iWvXruGTTz4p8p/ri/j7+6NBgwYICgrCxYsXcfz4cUyfPl2tzsCBA2FmZoagoCBcuXIFR44cwbhx4zB48GDp9Kc+fPHFF1i8eDG2bduGuLg4TJkyBTExMZgwYUKp282ZMwcLFy7Et99+ixs3buDy5cvYsGEDli9fDgBYvnw5fvrpJ1y/fh03btzAjh074OzsDDs7OwBPr8NFRUUhJSUFDx8+BPD0OmObNm3QuXNnrFq1ChcvXkR8fDy2b9+ON998E3/99VeROGrWrAm5XI7vvvsO8fHx2L17N+bNm6dWZ9asWfj1119x8+ZNXL16Fb/99hsaN26sUZzaGjNmDNLS0tC/f3+cOXMGt27dwv79+zFs2DCt58rs2LEjMjMzcfXqVals/fr1eOedd+Dl5YUmTZpIS9++fWFnZ4ctW7YAePp5kslkGDJkCM6dO4ebN28iPDwcoaGh+Pzzz6X9laXNgf+dAi1tKS0BDhgwAHK5HMOHD8fVq1exbds2rFixAsHBwVKdXbt2oVGjRmrbLVmyBJcvX8bVq1cxb948LFq0CN9++63aQKrt27fj2rVr0vuYOnUqAgMD1c6UHD9+HO3atZNOhVaIlxpDWgkZym0QlUVJQ8UXLlwoqlevLjIzM8W///4revbsKaysrISjo6OYMWOGGDJkiNp2zw/LF0KInj17iqCgIOl1XFycaNu2rZDL5aJBgwYiMjKyzLdBPKu4Y9eqVUt888030uvnj/OsgoICMXv2bOHm5iZMTU1LvA3i2WH+hbZs2SK8vb2FXC4X9vb2on379mLnzp1CCCHWrFkjvL29haWlpbCxsRGdO3dWG2q/e/duUa9ePWFiYiLdBiGEEDk5OWLhwoWiadOmUju0adNGbNy4UeTl5RX7nrdu3So8PDyEQqEQvr6+Yvfu3Woxz5s3TzRu3FiYm5uLatWqiZ49e4r4+HiN4ny+7by8vNRucSmufW7cuCF69+4t7OzshLm5uWjUqJH47LPPpFt0ivudlaRv375iypQpQgghUlJShImJidi+fXuxdUePHq12C09cXJzo3bu3cHV1FZaWlsLLy0usXbu2yK1CmrS5Ply8eFG0bdtWKBQK4ebmJhYtWqS2fsOGDeL5NNGxY0dha2srzMzMROvWrcXevXvV1kdERIjmzZsLKysrYWlpKTw9PcWCBQuKfD81bNhQ/PTTTyXGVh63QciEKOMV2EoqIyMDtra2SE9PR5rSGG8tPQprhQkuz6mYASO6kJOTg4SEBNSuXVtv04YQGapLly6hS5cuuHXrVrEDa0h7+/btw+eff45Lly6VeJq6tO+1Z7/HbWxsyhwHT4ESEZWiWbNmWLx4MRISEio6lCojKysLGzZseOlrtC+Lg2CIiF5g6NChFR1ClfLBBx9UdAgA2AMkIiIDxQRIREQGiQmwCjGw8UxEVIWVx/cZE2AVUHhvTUU+UoiISJcKv8/0Me1XoVdiEMyqVauwZMkSpKSkwMvLC999912pT8HYsWMHZs6cidu3b6N+/fpYvHgxunfvXo4Rv1qMjY1hZ2cnPXvSwsJC6wfyEhG9CoQQyM7OxoMHD2BnZ6f2tB1dq/AEuG3bNgQHByMsLAytW7dGaGgoAgICEBcXB0dHxyL1T548if79+2PhwoV49913sXXrVvTq1Qvnz5/X24zQlUHhs/uefwAzEVFl9PxMJfpQ4TfCt27dGm+88YY0/Y1KpYK7uzvGjRuHKVOmFKkfGBiIrKws/Pbbb1LZm2++CW9vb4SFhb3weFXxRvhnFRQUIC8vr6LDICIqM1NT01J7frq6Eb5Ce4BKpRLnzp3D1KlTpTIjIyP4+/vj1KlTxW5z6tQptWfVAU/nqPrll1+KrZ+bm4vc3FzptTZP36+MjI2N9XrKgIioqqjQQTCpqakoKCgo8qBhJycnpKSkFLtNSkqKVvUXLlwIW1tbaSl8Wj0RERm2Kj8KdOrUqUhPT5eWpKQkaZ2rnTmOTnoLe8a3q8AIiYioIlToKVAHBwcYGxsXOyljSRc/S5rEsaT6CoVCbS6uZ8lNjODh8OJJMYmIqOqp0AQol8vRokULREVFoVevXgCeDoKJiorC2LFji93G19cXUVFR+Oyzz6SygwcPwtfXV6NjFo75qerXAomIqqrC7++XHsP5UpMp6UBERIRQKBRi48aNIjY2VowcOVLY2dmJlJQUIYQQgwcPlubiEkKI6OhoYWJiIpYuXSquXbsmQkJChKmpqbh8+bJGx0tKShIAuHDhwoVLJV+SkpJeKv9U+H2AgYGB+OeffzBr1iykpKTA29sbkZGR0kCXxMREGBn971Kln58ftm7dihkzZmDatGmoX78+fvnlF43vAXR1dUVSUhKsra0hk8mQkZEBd3d3JCUlvdRw2qqK7fNibKPSsX1ejG1UuufbRwiBx48fw9XV9aX2W+H3AVY0Xd1PUlWxfV6MbVQ6ts+LsY1Kp6/2qfKjQImIiIrDBEhERAbJ4BOgQqFASEhIibdKGDq2z4uxjUrH9nkxtlHp9NU+Bn8NkIiIDJPB9wCJiMgwMQESEZFBYgIkIiKDxARIREQGySAS4KpVq+Dh4QEzMzO0bt0af/75Z6n1d+zYgUaNGsHMzAxNmzbF3r17yynSiqFN+6xduxbt2rWDvb097O3t4e/v/8L2rAq0/QwVioiIgEwmk551W1Vp2z6PHj3CmDFj4OLiAoVCgQYNGvDv7DmhoaFo2LAhzM3N4e7ujokTJyInJ6ecoi1fv//+O3r06AFXV1fIZLIS53d91tGjR9G8eXMoFArUq1cPGzdu1P7AL/UgtUogIiJCyOVyER4eLq5evSpGjBgh7OzsxP3794utHx0dLYyNjcXXX38tYmNjxYwZM7R61mhlo237DBgwQKxatUpcuHBBXLt2TQwdOlTY2tqKu3fvlnPk5UfbNiqUkJAg3NzcRLt27UTPnj3LJ9gKoG375ObmipYtW4ru3buLEydOiISEBHH06FERExNTzpGXH23baMuWLUKhUIgtW7aIhIQEsX//fuHi4iImTpxYzpGXj71794rp06eLnTt3CgBi165dpdaPj48XFhYWIjg4WMTGxorvvvtOGBsbi8jISK2OW+UTYKtWrcSYMWOk1wUFBcLV1VUsXLiw2Pp9+/YV77zzjlpZ69atxSeffKLXOCuKtu3zvPz8fGFtbS02bdqkrxArXFnaKD8/X/j5+Yl169aJoKCgKp0AtW2f1atXizp16gilUlleIVY4bdtozJgxolOnTmplwcHBok2bNnqN81WgSQL88ssvxeuvv65WFhgYKAICArQ6VpU+BapUKnHu3Dn4+/tLZUZGRvD398epU6eK3ebUqVNq9QEgICCgxPqVWVna53nZ2dnIy8tDtWrV9BVmhSprG82dOxeOjo4YPnx4eYRZYcrSPrt374avry/GjBkDJycnNGnSBAsWLEBBQUF5hV2uytJGfn5+OHfunHSaND4+Hnv37kX37t3LJeZXna6+pyt8Ngh9Sk1NRUFBgTSzRCEnJydcv3692G1SUlKKrZ+SkqK3OCtKWdrneZMnT4arq2uRD2NVUZY2OnHiBNavX4+YmJhyiLBilaV94uPjcfjwYQwcOBB79+7FzZs38emnnyIvLw8hISHlEXa5KksbDRgwAKmpqWjbti2EEMjPz8eoUaMwbdq08gj5lVfS93RGRgaePHkCc3NzjfZTpXuApF+LFi1CREQEdu3aBTMzs4oO55Xw+PFjDB48GGvXroWDg0NFh/NKUqlUcHR0xJo1a9CiRQsEBgZi+vTpCAsLq+jQXhlHjx7FggUL8P333+P8+fPYuXMn9uzZg3nz5lV0aFVKle4BOjg4wNjYGPfv31crv3//PpydnYvdxtnZWav6lVlZ2qfQ0qVLsWjRIhw6dAjNmjXTZ5gVSts2unXrFm7fvo0ePXpIZSqVCgBgYmKCuLg41K1bV79Bl6OyfIZcXFxgamoKY2Njqaxx48ZISUmBUqmEXC7Xa8zlrSxtNHPmTAwePBgff/wxAKBp06bIysrCyJEjMX36dLU5Ug1RSd/TNjY2Gvf+gCreA5TL5WjRogWioqKkMpVKhaioKPj6+ha7ja+vr1p9ADh48GCJ9SuzsrQPAHz99deYN28eIiMj0bJly/IItcJo20aNGjXC5cuXERMTIy3vvfceOnbsiJiYGLi7u5dn+HpXls9QmzZtcPPmTekfAwC4ceMGXFxcqlzyA8rWRtnZ2UWSXOE/DIKPb9bd97R243Mqn4iICKFQKMTGjRtFbGysGDlypLCzsxMpKSlCCCEGDx4spkyZItWPjo4WJiYmYunSpeLatWsiJCSkyt8GoU37LFq0SMjlcvHzzz+L5ORkaXn8+HFFvQW907aNnlfVR4Fq2z6JiYnC2tpajB07VsTFxYnffvtNODo6iq+++qqi3oLeadtGISEhwtraWvz0008iPj5eHDhwQNStW1f07du3ot6CXj1+/FhcuHBBXLhwQQAQy5cvFxcuXBB37twRQggxZcoUMXjwYKl+4W0QX3zxhbh27ZpYtWoVb4MoyXfffSdq1qwp5HK5aNWqlfjjjz+kdR06dBBBQUFq9bdv3y4aNGgg5HK5eP3118WePXvKOeLypU371KpVSwAosoSEhJR/4OVI28/Qs6p6AhRC+/Y5efKkaN26tVAoFKJOnTpi/vz5Ij8/v5yjLl/atFFeXp6YPXu2qFu3rjAzMxPu7u7i008/FQ8fPiz/wMvBkSNHiv1eKWyToKAg0aFDhyLbeHt7C7lcLurUqSM2bNig9XE5HRIRERmkKn0NkIiIqCRMgEREZJCYAImIyCAxARIRkUFiAiQiIoPEBEhERAaJCZCIiAwSEyARERkkJkAq0caNG2FnZ1fRYZSZTCbDL7/8UmqdoUOHolevXuUSz6tm5syZGDlyZLkc6+jRo5DJZHj06FGp9Tw8PBAaGqrXWLQ9hq7+DjT5PGorNjYWNWrUQFZWlk73ayiYAKu4oUOHQiaTFVlu3rxZ0aFh48aNUjxGRkaoUaMGhg0bhgcPHuhk/8nJyXj77bcBALdv34ZMJisyR9+KFSuwceNGnRyvJLNnz5bep7GxMdzd3TFy5EikpaVptR9dJuuUlBSsWLEC06dPV9t/YZxyuRz16tXD3LlzkZ+f/9LH8/PzQ3JyMmxtbQGUnFTOnDlTbkm5Mpg/fz78/PxgYWFRbHt5enrizTffxPLly8s/uCqACdAAdOvWDcnJyWpL7dq1KzosAICNjQ2Sk5Nx9+5drF27Fvv27cPgwYN1sm9nZ2coFIpS69ja2pZLL/f1119HcnIyEhMTsWHDBkRGRmL06NF6P25J1q1bBz8/P9SqVUutvPCz8tdff+Hzzz/H7NmzsWTJkpc+nlwuh7OzM2QyWan1qlevDgsLi5c+XlWhVCrx4YcflvpZGTZsGFavXq2Tf1QMDROgAVAoFHB2dlZbjI2NsXz5cjRt2hSWlpZwd3fHp59+iszMzBL3c/HiRXTs2BHW1tawsbFBixYtcPbsWWn9iRMn0K5dO5ibm8Pd3R3jx49/4akZmUwGZ2dnuLq64u2338b48eNx6NAhPHnyBCqVCnPnzkWNGjWgUCjg7e2NyMhIaVulUomxY8fCxcUFZmZmqFWrFhYuXKi278JTToUJ38fHBzKZDG+99RYA9V7VmjVr4OrqqjZNDwD07NkTH330kfT6119/RfPmzWFmZoY6depgzpw5L/zyMTExgbOzM9zc3ODv748PP/wQBw8elNYXFBRg+PDhqF27NszNzdGwYUOsWLFCWj979mxs2rQJv/76q9RLO3r0KAAgKSkJffv2hZ2dHapVq4aePXvi9u3bpcYTERGhNmdhocLPSq1atTB69Gj4+/tj9+7dAICHDx9iyJAhsLe3h4WFBd5++2389ddf0rZ37txBjx49YG9vD0tLS7z++uvYu3cvAPVToEePHsWwYcOQnp4uvZfZs2cDUD89OWDAAAQGBqrFl5eXBwcHB2zevBnA02mFFi5cKLWbl5cXfv7551Lf+/M0/Tv45ZdfUL9+fZiZmSEgIABJSUlq68vyuXiROXPmYOLEiWjatGmJdbp06YK0tDQcO3bspY5liJgADZiRkRG+/fZbXL16FZs2bcLhw4fx5Zdfllh/4MCBqFGjBs6cOYNz585hypQpMDU1BfB0Ithu3brh/fffx6VLl7Bt2zacOHECY8eO1Somc3NzqFQq5OfnY8WKFVi2bBmWLl2KS5cuISAgAO+99570pfvtt99i9+7d2L59O+Li4rBlyxZ4eHgUu98///wTAHDo0CEkJydj586dRep8+OGH+Pfff3HkyBGpLC0tDZGRkRg4cCAA4Pjx4xgyZAgmTJiA2NhY/PDDD9i4cSPmz5+v8Xu8ffs29u/frzb3nUqlQo0aNbBjxw7ExsZi1qxZmDZtGrZv3w4AmDRpEvr27avWm/fz80NeXh4CAgJgbW2N48ePIzo6GlZWVujWrRuUSmWxx09LS0NsbKxGczmam5tL+xk6dCjOnj2L3bt349SpUxBCoHv37sjLywMAjBkzBrm5ufj9999x+fJlLF68GFZWVkX26efnh9DQUKn3n5ycjEmTJhWpN3DgQPz3v/9VS0b79+9HdnY2evfuDQBYuHAhNm/ejLCwMFy9ehUTJ07EoEGDtEoGmvwdZGdnY/78+di8eTOio6Px6NEj9OvXT1pfls/FW2+9haFDh2ocZ0nkcjm8vb1x/Pjxl96XwXnJWSzoFRcUFCSMjY2FpaWltHzwwQfF1t2xY4d47bXXpNcbNmwQtra20mtra2uxcePGYrcdPny4GDlypFrZ8ePHhZGRkXjy5Emx2zy//xs3bogGDRqIli1bCiGEcHV1FfPnz1fb5o033hCffvqpEEKIcePGiU6dOgmVSlXs/gGIXbt2CSGESEhIEADEhQsX1Oo8P1VRz549xUcffSS9/uGHH4Srq6soKCgQQgjRuXNnsWDBArV9/Pjjj8LFxaXYGIR4OrebkZGRsLS0FGZmZtJUL8uXLy9xGyGEGDNmjHj//fdLjLXw2A0bNlRrg9zcXGFubi72799f7H4L51xLTExUK392/yqVShw8eFAoFAoxadIkcePGDQFAREdHS/VTU1OFubm52L59uxBCiKZNm4rZs2cXe8zC6W4Kp/N5/ndfqFatWuKbb74RQjydEsjBwUFs3rxZWt+/f38RGBgohBAiJydHWFhYiJMnT6rtY/jw4aJ///7FxvH8MYpT3N8BALXpi65duyYAiNOnTwshNPtcPPt5FOLF80g+q6T2KtS7d28xdOhQjfZF/2NSUYmXyk/Hjh2xevVq6bWlpSWAp72hhQsX4vr168jIyEB+fj5ycnKQnZ1d7HWY4OBgfPzxx/jxxx+l03h169YF8PT06KVLl7BlyxapvhACKpUKCQkJaNy4cbGxpaenw8rKCiqVCjk5OWjbti3WrVuHjIwM/P3332jTpo1a/TZt2uDixYsAnvZIunTpgoYNG6Jbt25499130bVr15dqq4EDB2LEiBH4/vvvoVAosGXLFvTr10+anfvixYuIjo5W+8++oKCg1HYDgIYNG2L37t3IycnB//3f/yEmJgbjxo1Tq7Nq1SqEh4cjMTERT548gVKphLe3d6nxXrx4ETdv3oS1tbVaeU5ODm7dulXsNk+ePAEAmJmZFVn322+/wcrKCnl5eVCpVBgwYABmz56NqKgomJiYoHXr1lLd1157DQ0bNsS1a9cAAOPHj8fo0aNx4MAB+Pv74/3330ezZs1Kjb80JiYm6Nu3L7Zs2YLBgwcjKysLv/76KyIiIgAAN2/eRHZ2Nrp06aK2nVKphI+Pj8bH0eTvwMTEBG+88Ya0TaNGjWBnZ4dr166hVatWZfpcFJ7G1QVzc3NkZ2frbH+GggnQAFhaWqJevXpqZbdv38a7776L0aNHY/78+ahWrRpOnDiB4cOHQ6lUFvsHO3v2bAwYMAB79uzBvn37EBISgoiICPTu3RuZmZn45JNPMH78+CLb1axZs8TYrK2tcf78eRgZGcHFxQXm5uYAgIyMjBe+r+bNmyMhIQH79u3DoUOH0LdvX/j7+2t9DehZPXr0gBACe/bswRtvvIHjx4/jm2++kdZnZmZizpw56NOnT5Fti0sohQpHVQLAokWL8M4772DOnDmYN28egKfX5CZNmoRly5bB19cX1tbWWLJkCU6fPl1qvJmZmWjRooXaPx6FqlevXuw2Dg4OAJ5e03u+TuE/S3K5HK6urjAx0fwr4uOPP0ZAQAD27NmDAwcOYOHChVi2bFmRRK+NgQMHokOHDnjw4AEOHjwIc3NzdOvWDQCkU6N79uyBm5ub2nYvGvxUqCx/B8Up6+dCV9LS0qR/RklzTIAG6ty5c1CpVFi2bJnUuym83lSaBg0aoEGDBpg4cSL69++PDRs2oHfv3mjevDliY2OLJNoXMTIyKnYbGxsbuLq6Ijo6Gh06dJDKo6Oj0apVK7V6gYGBCAwMxAcffIBu3bohLS0N1apVU9tf4fW2goKCUuMxMzNDnz59sGXLFty8eRMNGzZE8+bNpfXNmzdHXFyc1u/zeTNmzECnTp0wevRo6X36+fnh008/leo834OTy+VF4m/evDm2bdsGR0dH2NjYaHTsunXrwsbGBrGxsWjQoIHauuL+WQKAxo0bIz8/H6dPn4afnx8A4N9//0VcXBw8PT2leu7u7hg1ahRGjRqFqVOnYu3atcUmwOLeS3H8/Pzg7u6Obdu2Yd++ffjwww+l686enp5QKBRITExU+4xoQ9O/g/z8fJw9e1b67MXFxeHRo0fSmQ1dfS7K6sqVK/jggw8q5NiVGQfBGKh69eohLy8P3333HeLj4/Hjjz8iLCysxPpPnjzB2LFjcfToUdy5cwfR0dE4c+aM9AUwefJknDx5EmPHjkVMTAz++usv/Prrr1oPgnnWF198gcWLF2Pbtm2Ii4vDlClTEBMTgwkTJgB4Onrvp59+wvXr13Hjxg3s2LEDzs7Oxd7W4OjoCHNzc0RGRuL+/ftIT08v8bgDBw7Enj17EB4eLg1+KTRr1ixs3rwZc+bMwdWrV3Ht2jVERERgxowZWr03X19fNGvWDAsWLAAA1K9fH2fPnsX+/ftx48YNzJw5E2fOnFHbxsPDA5cuXUJcXBxSU1ORl5eHgQMHwsHBAT179sTx48eRkJCAo0ePYvz48bh7926xxzYyMoK/vz9OnDihcbz169dHz549MWLECJw4cQIXL17EoEGD4Obmhp49ewIAPvvsM+zfvx8JCQk4f/48jhw5UuKpbw8PD2RmZiIqKgqpqamlnr4bMGAAwsLCcPDgQbXfh7W1NSZNmoSJEydi06ZNuHXrFs6fP4/vvvsOmzZt0uh9afp3YGpqinHjxuH06dM4d+4chg4dijfffFNKiGX5XAwZMgRTp04tNb7ExETExMQgMTERBQUFiImJQUxMjNrAoNu3b+PevXvw9/fX6D3TMyr6IiTpV3EDJwotX75cuLi4CHNzcxEQECA2b95c4kCF3Nxc0a9fP+Hu7i7kcrlwdXUVY8eOVRvg8ueff4ouXboIKysrYWlpKZo1a1ZkEMuzXnRhv6CgQMyePVu4ubkJU1NT4eXlJfbt2yetX7NmjfD29haWlpbCxsZGdO7cWZw/f15aj+cGHaxdu1a4u7sLIyMj0aFDhxLbp6CgQLi4uAgA4tatW0XiioyMFH5+fsLc3FzY2NiIVq1aiTVr1pT4PkJCQoSXl1eR8p9++kkoFAqRmJgocnJyxNChQ4Wtra2ws7MTo0ePFlOmTFHb7sGDB1L7AhBHjhwRQgiRnJwshgwZIhwcHIRCoRB16tQRI0aMEOnp6SXGtHfvXuHm5iYN7impLZ6VlpYmBg8eLGxtbaXPzI0bN6T1Y8eOFXXr1hUKhUJUr15dDB48WKSmpgohig6CEUKIUaNGiddee00AECEhIUKI4geoxMbGCgCiVq1aRQY8qVQqERoaKho2bChMTU1F9erVRUBAgDh27FiJ7+P5Y2j6d/Cf//xH1KlTRygUCuHv7y/u3Lmjtt8XfS6e/zx26NBBBAUFlRinEE9/J/j/g6aeXQp/90IIsWDBAhEQEFDqfqh4MiGEqIjES0QVRwiB1q1bS6eyqXJSKpWoX78+tm7dWmTAGL0YT4ESGSCZTIY1a9bw6SGVXGJiIqZNm8bkV0bsARIRkUFiD5CIiAwSEyARERkkJkAiIjJITIBERGSQmACJiMggMQESEZFBYgIkIiKDxARIREQGiQmQiIgM0v8Dl0CDLsNxUoEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import RocCurveDisplay\n",
    "roc_curve_display = RocCurveDisplay.from_estimator(estimator=clf, \n",
    "                                                   X=X_test, \n",
    "                                                   y=y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most ideal position for a ROC curve to run along the top left corner of the plot. \n",
    "\n",
    "This would mean the model predicts only true positives and no false positives. And would result in a ROC AUC score of 1.0.\n",
    "\n",
    "You can see this by creating a ROC curve using only the `y_test` labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB1BUlEQVR4nO3dd1hT1/8H8HcCJGxQEURFEfdGUKl7obhQWxUcVRzV1l2te6GtitY669YqbsHR1ol111W1ItaFAwcOQHGwZCbn94c/8jUCShC4jPfrefJoTs69eecSyCfnnnuvTAghQERERFRAyKUOQERERJSdWNwQERFRgcLihoiIiAoUFjdERERUoLC4ISIiogKFxQ0REREVKCxuiIiIqEBhcUNEREQFCosbIiIiKlBY3FCus7e3R79+/aSOUeg0b94czZs3lzrGJ82YMQMymQyRkZFSR8lzZDIZZsyYkS3revjwIWQyGXx9fbNlfQBw8eJFKBQKPHr0KNvWmd169OgBDw8PqWNQDmNxU8D4+vpCJpNpbvr6+ihVqhT69euHp0+fSh0vT4uLi8NPP/2EWrVqwdjYGBYWFmjSpAk2bdqE/HKVkps3b2LGjBl4+PCh1FHSUKlU2LBhA5o3b46iRYtCqVTC3t4e/fv3x7///it1vGyxbds2LF68WOoYWnIz05QpU9CzZ0+ULVtW09a8eXOtv0lGRkaoVasWFi9eDLVane56Xr58iXHjxqFy5cowNDRE0aJF4ebmhv3792f43NHR0Zg5cyZq164NU1NTGBkZoUaNGpgwYQKePXum6TdhwgTs3r0bV69ezfTrKgzv3QJHUIGyYcMGAUD8+OOPYvPmzWLt2rVi4MCBQk9PT5QvX17Ex8dLHVEkJCSIpKQkqWNoCQ8PF9WrVxdyuVz06tVLrF69WixZskQ0bdpUABCenp4iJSVF6piftHPnTgFAnDhxIs1jiYmJIjExMfdDCSHevn0r2rZtKwCIpk2bivnz54vffvtNTJs2TVSuXFnIZDLx+PFjIYQQ3t7eAoB48eKFJFk/R4cOHUTZsmVzbP3x8fEiOTlZp2UyyqRWq0V8fHy2va+vXLkiAIhz585ptTdr1kyULl1abN68WWzevFksWrRI1KtXTwAQkydPTrOe4OBgUapUKaFQKMS3334r1q5dK+bPny8cHR0FADF27Ng0y4SEhIhy5coJPT090aNHD7Fs2TKxZs0aMXz4cFGsWDFRsWJFrf7169cXffr0ydTr0uW9S3kHi5sCJrW4uXTpklb7hAkTBADh5+cnUTJpxcfHC5VKleHjbm5uQi6Xiz///DPNY2PHjhUAxNy5c3MyYrpiY2N16v+x4kZKw4YNEwDEokWL0jyWkpIi5s+fn6vFjVqtFm/fvs329eZEcaNSqT7rS0lOF1ypRo4cKcqUKSPUarVWe7NmzUT16tW12uLj40XZsmWFmZmZVnGVlJQkatSoIYyNjcU///yjtUxKSorw9PQUAMSOHTs07cnJyaJ27drC2NhYnD59Ok2uqKioNEXUL7/8IkxMTERMTMwnX5cu793P8bk/Z9LG4qaAyai42b9/vwAg5syZo9V+69Yt0bVrV1GkSBGhVCqFs7Nzuh/wr1+/Ft9//70oW7asUCgUolSpUqJPnz5aH0AJCQli+vTponz58kKhUIjSpUuLcePGiYSEBK11lS1bVnh5eQkhhLh06ZIAIHx9fdM8Z0BAgAAg9u3bp2l78uSJ6N+/v7C2thYKhUJUq1ZN/Pbbb1rLnThxQgAQ27dvF1OmTBElS5YUMplMvH79Ot1tdv78eQFADBgwIN3Hk5OTRcWKFUWRIkU0H4gPHjwQAMT8+fPFwoULRZkyZYShoaFo2rSpuHbtWpp1ZGY7p/7sTp48KYYMGSKKFy8uLC0thRBCPHz4UAwZMkRUqlRJGBoaiqJFi4pu3bqJBw8epFn+w1tqodOsWTPRrFmzNNvJz89PzJo1S5QqVUoolUrRsmVLcffu3TSvYdmyZaJcuXLC0NBQ1KtXT/z9999p1pmex48fC319fdG6deuP9kuVWtzcvXtXeHl5CQsLC2Fubi769esn4uLitPquX79etGjRQhQvXlwoFApRtWpVsWLFijTrLFu2rOjQoYMICAgQzs7OQqlUaj6sMrsOIYQ4ePCgaNq0qTA1NRVmZmaibt26YuvWrUKId9v3w23/flGR2d8PAGLYsGFiy5Ytolq1akJfX1/8/vvvmse8vb01faOjo8WoUaM0v5fFixcXrq6u4vLly5/MlPoe3rBhg9bz37p1S3Tv3l1YWVkJQ0NDUalSpXRHWD5UpkwZ0a9fvzTt6RU3QgjRrVs3AUA8e/ZM07Z9+3bNyHN63rx5IywtLUWVKlU0bTt27BAAxOzZsz+ZMdXVq1cFALFnz56P9tP1vevl5ZVuIZn6nn5fej9nf39/UaRIkXS3Y1RUlFAqleKHH37QtGX2PVUY6Wf7fi7Kk1LnYBQpUkTTduPGDTRq1AilSpXCxIkTYWJiAn9/f3Tp0gW7d+/Gl19+CQCIjY1FkyZNcOvWLQwYMABOTk6IjIzE3r178eTJE1hZWUGtVqNTp044c+YMBg8ejKpVq+LatWtYtGgR7ty5gz/++CPdXHXr1oWDgwP8/f3h5eWl9Zifnx+KFCkCNzc3AEBERAS++OILyGQyDB8+HMWLF8ehQ4cwcOBAREdH4/vvv9da/qeffoJCocDYsWORmJgIhUKRboZ9+/YBAPr27Zvu4/r6+ujVqxdmzpyJs2fPwtXVVfPYpk2bEBMTg2HDhiEhIQFLlixBy5Ytce3aNdjY2Oi0nVMNHToUxYsXx/Tp0xEXFwcAuHTpEs6dO4cePXqgdOnSePjwIVauXInmzZvj5s2bMDY2RtOmTTFy5EgsXboUkydPRtWqVQFA829G5s6dC7lcjrFjxyIqKgo///wzevfujQsXLmj6rFy5EsOHD0eTJk0wevRoPHz4EF26dEGRIkVQunTpj67/0KFDSElJQZ8+fT7a70MeHh4oV64cfHx8EBgYiHXr1sHa2hrz5s3TylW9enV06tQJ+vr62LdvH4YOHQq1Wo1hw4Zpre/27dvo2bMnvv32WwwaNAiVK1fWaR2+vr4YMGAAqlevjkmTJsHS0hJXrlxBQEAAevXqhSlTpiAqKgpPnjzBokWLAACmpqYAoPPvx/Hjx+Hv74/hw4fDysoK9vb26W6j7777Drt27cLw4cNRrVo1vHz5EmfOnMGtW7fg5OT00Uzp+e+//9CkSRMYGBhg8ODBsLe3R0hICPbt24fZs2dnuNzTp08RGhoKJyenDPt8KHVCs6WlpabtU7+LFhYW6Ny5MzZu3Ih79+6hQoUK2Lt3LwDo9P6qVq0ajIyMcPbs2TS/f+/L6ns3sz78OVesWBFffvkl9uzZg9WrV2v9zfrjjz+QmJiIHj16AND9PVXoSF1dUfZK/fZ+9OhR8eLFC/H48WOxa9cuUbx4caFUKrWGT1u1aiVq1qypVeWr1WrRsGFDrX3U06dPz/BbTuoQ9ObNm4VcLk8zLLxq1SoBQJw9e1bT9v7IjRBCTJo0SRgYGIhXr15p2hITE4WlpaXWaMrAgQOFra2tiIyM1HqOHj16CAsLC82oSuqIhIODQ6Z2PXTp0kUAyHBkRwgh9uzZIwCIpUuXCiH+963XyMhIPHnyRNPvwoULAoAYPXq0pi2z2zn1Z9e4ceM08yDSex2pI06bNm3StH1st1RGIzdVq1bVmouzZMkSAUAzApWYmCiKFSsm6tWrpzXfw9fXVwD45MjN6NGjBQBx5cqVj/ZLlfot98ORtC+//FIUK1ZMqy297eLm5iYcHBy02sqWLSsAiICAgDT9M7OON2/eCDMzM+Hi4pJm18H7u2Ey2gWky+8HACGXy8WNGzfSrAcfjNxYWFiIYcOGpen3vowypTdy07RpU2FmZiYePXqU4WtMz9GjR9OMsqZq1qyZqFKlinjx4oV48eKFCA4OFuPGjRMARIcOHbT6Ojo6CgsLi48+18KFCwUAsXfvXiGEEHXq1PnkMumpVKmSaNeu3Uf76Pre1XXkJr2f8+HDh9Pdlu3bt9d6T+ryniqMeLRUAeXq6orixYvDzs4O3bp1g4mJCfbu3av5lv3q1SscP34cHh4eiImJQWRkJCIjI/Hy5Uu4ubnh7t27mqOrdu/ejdq1a6f7DUcmkwEAdu7ciapVq6JKlSqadUVGRqJly5YAgBMnTmSY1dPTE8nJydizZ4+m7a+//sKbN2/g6ekJABBCYPfu3XB3d4cQQus53NzcEBUVhcDAQK31enl5wcjI6JPbKiYmBgBgZmaWYZ/Ux6Kjo7Xau3TpglKlSmnu169fHy4uLjh48CAA3bZzqkGDBkFPT0+r7f3XkZycjJcvX6JChQqwtLRM87p11b9/f61viE2aNAEA3L9/HwDw77//4uXLlxg0aBD09f832Nu7d2+tkcCMpG6zj23f9Hz33Xda95s0aYKXL19q/Qze3y5RUVGIjIxEs2bNcP/+fURFRWktX65cOc0o4Psys44jR44gJiYGEydOhKGhodbyqb8DH6Pr70ezZs1QrVq1T67X0tISFy5c0DoaKKtevHiBv//+GwMGDECZMmW0HvvUa3z58iUAZPh+CA4ORvHixVG8eHFUqVIF8+fPR6dOndIchh4TE/PJ98mHv4vR0dE6v7dSs37qdANZfe9mVno/55YtW8LKygp+fn6attevX+PIkSOav4fA5/3NLQy4W6qAWr58OSpVqoSoqCisX78ef//9N5RKpebxe/fuQQiBadOmYdq0aemu4/nz5yhVqhRCQkLQtWvXjz7f3bt3cevWLRQvXjzDdWWkdu3aqFKlCvz8/DBw4EAA73ZJWVlZaX5RX7x4gTdv3mDNmjVYs2ZNpp6jXLlyH82cKvUPV0xMjNYQ+fsyKoAqVqyYpm+lSpXg7+8PQLft/LHc8fHx8PHxwYYNG/D06VOtQ9M//BDX1YcfZKkfUK9fvwYAzTlLKlSooNVPX18/w90l7zM3Nwfwv22YHblS13n27Fl4e3vj/PnzePv2rVb/qKgoWFhYaO5n9H7IzDpCQkIAADVq1NDpNaTS9fcjs+/dn3/+GV5eXrCzs4OzszPat2+Pvn37wsHBQeeMqcVsVl8jgAxPmWBvb4+1a9dCrVYjJCQEs2fPxosXL9IUimZmZp8sOD78XTQ3N9dk1zXrp4q2rL53Myu9n7O+vj66du2Kbdu2ITExEUqlEnv27EFycrJWcfM5f3MLAxY3BVT9+vVRt25dAO9GFxo3boxevXrh9u3bMDU11ZxfYuzYsel+mwXSfph9jFqtRs2aNbFw4cJ0H7ezs/vo8p6enpg9ezYiIyNhZmaGvXv3omfPnpqRgtS8X3/9dZq5Oalq1aqldT8zozbAuzkpf/zxB/777z80bdo03T7//fcfAGTq2/T7srKd08s9YsQIbNiwAd9//z0aNGgACwsLyGQy9OjRI8NzhWTWh6NEqTL6oNJVlSpVAADXrl2Do6Njppf7VK6QkBC0atUKVapUwcKFC2FnZweFQoGDBw9i0aJFabZLettV13Vkla6/H5l973p4eKBJkyb4/fff8ddff2H+/PmYN28e9uzZg3bt2n127swqVqwYgP8VxB8yMTHRmqvWqFEjODk5YfLkyVi6dKmmvWrVqggKCkJoaGia4jbVh7+LVapUwZUrV/D48eNP/p153+vXr9P9cvI+Xd+7GRVLKpUq3faMfs49evTA6tWrcejQIXTp0gX+/v6oUqUKateurenzuX9zCzoWN4WAnp4efHx80KJFCyxbtgwTJ07UfLMzMDDQ+qOTnvLly+P69euf7HP16lW0atUqU8P0H/L09MTMmTOxe/du2NjYIDo6WjNxDgCKFy8OMzMzqFSqT+bVVceOHeHj44NNmzalW9yoVCps27YNRYoUQaNGjbQeu3v3bpr+d+7c0Yxo6LKdP2bXrl3w8vLCggULNG0JCQl48+aNVr+sbPtPST0h271799CiRQtNe0pKCh4+fJimqPxQu3btoKenhy1btmTrxMx9+/YhMTERe/fu1fog1GU4PrPrKF++PADg+vXrHy36M9r+n/v78TG2trYYOnQohg4diufPn8PJyQmzZ8/WFDeZfb7U9+qnftfTk1oEPHjwIFP9a9Wqha+//hqrV6/G2LFjNdu+Y8eO2L59OzZt2oSpU6emWS46Ohp//vknqlSpovk5uLu7Y/v27diyZQsmTZqUqedPSUnB48eP0alTp4/20/W9W6RIkTS/kwB0PmNz06ZNYWtrCz8/PzRu3BjHjx/HlClTtPrk5HuqIOCcm0KiefPmqF+/PhYvXoyEhARYW1ujefPmWL16NcLCwtL0f/Hiheb/Xbt2xdWrV/H777+n6Zf6LdrDwwNPnz7F2rVr0/SJj4/XHPWTkapVq6JmzZrw8/ODn58fbG1ttQoNPT09dO3aFbt37073j+/7eXXVsGFDuLq6YsOGDemeAXXKlCm4c+cOxo8fn+ab1h9//KE1Z+bixYu4cOGC5oNFl+38MXp6emlGUn799dc03whNTEwAIN0/sFlVt25dFCtWDGvXrkVKSoqmfevWrRl+U3+fnZ0dBg0ahL/++gu//vprmsfVajUWLFiAJ0+e6JQrdWTnw110GzZsyPZ1tGnTBmZmZvDx8UFCQoLWY+8va2Jiku5uws/9/UiPSqVK81zW1tYoWbIkEhMTP5npQ8WLF0fTpk2xfv16hIaGaj32qVG8UqVKwc7OTqez9Y4fPx7JyclaIw/dunVDtWrVMHfu3DTrUqvVGDJkCF6/fg1vb2+tZWrWrInZs2fj/PnzaZ4nJiYmTWFw8+ZNJCQkoGHDhh/NqOt7t3z58oiKitKMLgFAWFhYun87P0Yul6Nbt27Yt28fNm/ejJSUFK1dUkDOvKcKEo7cFCLjxo1D9+7d4evri++++w7Lly9H48aNUbNmTQwaNAgODg6IiIjA+fPn8eTJE83pyceNG4ddu3ahe/fuGDBgAJydnfHq1Svs3bsXq1atQu3atdGnTx/4+/vju+++w4kTJ9CoUSOoVCoEBwfD398fhw8f1uwmy4inpyemT58OQ0NDDBw4EHK5du09d+5cnDhxAi4uLhg0aBCqVauGV69eITAwEEePHsWrV6+yvG02bdqEVq1aoXPnzujVqxeaNGmCxMRE7NmzBydPnoSnpyfGjRuXZrkKFSqgcePGGDJkCBITE7F48WIUK1YM48eP1/TJ7Hb+mI4dO2Lz5s2wsLBAtWrVcP78eRw9elSzOyCVo6Mj9PT0MG/ePERFRUGpVKJly5awtrbO8rZRKBSYMWMGRowYgZYtW8LDwwMPHz6Er68vypcvn6lvjQsWLEBISAhGjhyJPXv2oGPHjihSpAhCQ0Oxc+dOBAcHa43UZUabNm2gUCjg7u6Ob7/9FrGxsVi7di2sra3TLSQ/Zx3m5uZYtGgRvvnmG9SrVw+9evVCkSJFcPXqVbx9+xYbN24EADg7O8PPzw9jxoxBvXr1YGpqCnd392z5/fhQTEwMSpcujW7dumkuOXD06FFcunRJa4Qvo0zpWbp0KRo3bgwnJycMHjwY5cqVw8OHD3HgwAEEBQV9NE/nzp3x+++/Z2ouC/But1L79u2xbt06TJs2DcWKFYNCocCuXbvQqlUrNG7cGP3790fdunXx5s0bbNu2DYGBgfjhhx+03isGBgbYs2cPXF1d0bRpU3h4eKBRo0YwMDDAjRs3NKOu7x/KfuTIERgbG6N169afzKnLe7dHjx6YMGECvvzyS4wcORJv377FypUrUalSJZ0n/nt6euLXX3+Ft7c3atasmeaUDjnxnipQcv8ALcpJGZ3ET4h3Z8AsX768KF++vOZQ45CQENG3b19RokQJYWBgIEqVKiU6duwodu3apbXsy5cvxfDhwzWnRS9durTw8vLSOiw7KSlJzJs3T1SvXl0olUpRpEgR4ezsLGbOnCmioqI0/T48FDzV3bt3NScaO3PmTLqvLyIiQgwbNkzY2dkJAwMDUaJECdGqVSuxZs0aTZ/UQ5x37typ07aLiYkRM2bMENWrVxdGRkbCzMxMNGrUSPj6+qY5FPb9k/gtWLBA2NnZCaVSKZo0aSKuXr2aZt2Z2c4f+9m9fv1a9O/fX1hZWQlTU1Ph5uYmgoOD092Wa9euFQ4ODkJPTy9TJ/H7cDtldHK3pUuXirJlywqlUinq168vzp49K5ydnUXbtm0zsXXfnc113bp1okmTJsLCwkIYGBiIsmXLiv79+2sdapvRGYpTt8/7Jy7cu3evqFWrljA0NBT29vZi3rx5Yv369Wn6pZ7ELz2ZXUdq34YNGwojIyNhbm4u6tevL7Zv3655PDY2VvTq1UtYWlqmOYlfZn8/8P8nd0sP3jsUPDExUYwbN07Url1bmJmZCRMTE1G7du00JyDMKFNGP+fr16+LL7/8UlhaWgpDQ0NRuXJlMW3atHTzvC8wMFAASHNockYn8RNCiJMnT6Y5vF0IIZ4/fy7GjBkjKlSoIJRKpbC0tBSurq6aw7/T8/r1azF9+nRRs2ZNYWxsLAwNDUWNGjXEpEmTRFhYmFZfFxcX8fXXX3/yNaXK7HtXCCH++usvUaNGDaFQKETlypXFli1bPnoSv4yo1WphZ2cnAIhZs2al2yez76nCSCZEPrkiIFEe8vDhQ5QrVw7z58/H2LFjpY4jCbVajeLFi+Orr75Kd2icCp9WrVqhZMmS2Lx5s9RRMhQUFAQnJycEBgbqNMGd8hfOuSGiT0pISEgz72LTpk149eoVmjdvLk0oynPmzJkDPz8/nSfQ5qa5c+eiW7duLGwKOM65IaJP+ueffzB69Gh0794dxYoVQ2BgIH777TfUqFED3bt3lzoe5REuLi5ISkqSOsZH7dixQ+oIlAtY3BDRJ9nb28POzg5Lly7Fq1evULRoUfTt2xdz587N8JpdRERS4ZwbIiIiKlA454aIiIgKFBY3REREVKAUujk3arUaz549g5mZGU9ZTURElE8IIRATE4OSJUumOcnrhwpdcfPs2bNCf0ExIiKi/Orx48coXbr0R/sUuuLGzMwMwLuNk3o5eyIiIsrboqOjYWdnp/kc/5hCV9yk7ooyNzdncUNERJTPZGZKCScUExERUYHC4oaIiIgKFBY3REREVKCwuCEiIqIChcUNERERFSgsboiIiKhAYXFDREREBQqLGyIiIipQWNwQERFRgcLihoiIiAoUSYubv//+G+7u7ihZsiRkMhn++OOPTy5z8uRJODk5QalUokKFCvD19c3xnERERJR/SFrcxMXFoXbt2li+fHmm+j948AAdOnRAixYtEBQUhO+//x7ffPMNDh8+nMNJiYiIKL+Q9MKZ7dq1Q7t27TLdf9WqVShXrhwWLFgAAKhatSrOnDmDRYsWwc3NLadiZp4QgOqt1CmIiIikp2cMZOIilzkhX10V/Pz583B1ddVqc3Nzw/fff5/hMomJiUhMTNTcj46OzplwQgBHGgOR53Jm/URERPmJRyygbyLJU+erCcXh4eGwsbHRarOxsUF0dDTi4+PTXcbHxwcWFhaam52dXc6EU71lYUNERIVSZIwxnkdJU8ikJ1+N3GTFpEmTMGbMGM396OjonCtwUn0VIVm1SkRElJv+Pv0MPccdRdUqljh8sCP09P5/3ETPWLJM+aq4KVGiBCIiIrTaIiIiYG5uDiMjo3SXUSqVUCqVuRHvf/RNWNwQEVGBplYL+PhcwPTpZ6FWC5ibK/H8lQy2ttJ//uWr4qZBgwY4ePCgVtuRI0fQoEEDiRIREREVPhERcejT5yCOHHkEAOjbtxqWL3eFqalC4mTvSDrnJjY2FkFBQQgKCgLw7lDvoKAghIaGAni3S6lv376a/t999x3u37+P8ePHIzg4GCtWrIC/vz9Gjx4tRXwiIqJC5/jxUDg6bsKRI49gbKwPX9+22LixfZ4pbACJR27+/fdftGjRQnM/dW6Ml5cXfH19ERYWpil0AKBcuXI4cOAARo8ejSVLlqB06dJYt25d3jgMnIiIqIBLSVFj+PCjCA+PQ/XqxeDv745q1aykjpWGTAghpA6Rm6Kjo2FhYYGoqCiYm5tn34pT4gB/03f/l/DwNyIiopx09epzrFp1FQsWNIexsUGuPa8un9/56lBwIiIiyl1//fUQa9f+p7lfu7Y1Vq5snauFja7y1YRiIiIiyh0pKWp4e5+Fj88F6OvL4exsAycnm08vmAewuCEiIiItT57EoGfP/Thz5ikAYODAmqhWrZjEqTKPxQ0RERFpHDx4H337HsLLl/EwM1Ng3bo28PCoInUsnbC4ISIiIgDAlCmnMWfOBQCAk5MN/P3dUb68pbShsoATiomIiAgAULSoIQBgxIg6OHeuZ74sbACO3BARERVqcXFJMDF5dwK+MWPqwsXFFo0bl5Y41efhyA0REVEhlJSkwvffH0fdulsQG5sEAJDJZPm+sAFY3BARERU69++/QaNG27BkSSCCg19h374QqSNlK+6WIiIiKkR2776DAQMCEB2dhCJFDLFxYzu4u5eXOla2YnFDRERUCCQkpGDs2JNYvjwIANCwYUls394RZcpk46WI8gjuliIiIioExo07pSlsJkyoj5MnPQtkYQNw5IaIiKhQmDLlC5w8+Rjz5zdD27blpI6TozhyQ0REVADFxydj27ZbmvslSpjg6lWvAl/YABy5ISIiKnCCg1/Cw2Mfrl2LhL6+THP5BLlcJnGy3MHihoiIqADZtOkGhgw5grdvU2BtbYyiRY2kjpTrWNwQEREVAHFxSRgx4jg2bLgOAGjZsgy2bGkPW1tTiZPlPhY3RERE+dyNG5Hw8NiHmzdfQi6Xwdu7AaZM+QJ6eoVzai2LGyIionwuJOQNbt58CVtbE2zb1gHNm5eROpKkWNwQERHlQ0IIyGTvJgh36lQB69a5wd3dAdbWJhInk17hHK8iIiLKx65efY7Gjbfj8eNoTdvAgTVZ2Pw/FjdERET5hBACq1dfhYvLVpw79ww//HBS6kh5EndLERER5QPR0YkYPPgv+PndBgB06OCAFStcJU6VN7G4ISIiyuMCAyPg6bkP9+69gb6+HD4+TTBmTN1Cc1I+XbG4ISIiysNOnAhF27a7kZSkQpkyZvDzc8cXX5SUOlaexuKGiIgoD/viC1tUrlwEDg6WWL/erVCecVhXLG6IiIjymBs3IlGlSlHo6clhZGSAEyc8UbSooebQb/o4Hi1FRESURwghsGjRv6hTZxN8fC5o2osVM2JhowOO3BAREeUBr17Fo1+/AOzbFwIAuH49UutEfZR5LG6IiIgkdu7cU/TosR+PH8dAodDDokXNMWSIIwubLGJxQ0REJBG1WuCXXy5h8uTTUKkEKlSwhL+/O+rUsZE6Wr7G4oaIiEgiISFvMH36WahUAj17VsHq1W1gZqaQOla+x+KGiIhIIhUrFsGyZa0gBPDNNzW5GyqbsLghIiLKJWq1wNy5F+DqWhb169sCAL75ppbEqQoeHgpORESUCyIi4tC27S5MmXIGnp77EBeXJHWkAosjN0RERDns+PFQ9O59AOHhcTAy0oe3d0OYmHBuTU5hcUNERJRDVCo1fvrpPH788TyEAKpXLwZ/f3dUq2YldbQCjcUNERFRDoiOTkTnzn/g5MnHAIABA2rg119bwdjYQOJkBR+LGyIiohxgaqqAiYkBTEwMsGpVa3z9dTWpIxUaLG6IiIiySUqKGsnJKhgZGUAul2HjxnaIjIxH5cpFpY5WqPBoKSIiomzw5EkMWrb0x3ffHdG0FStmxMJGAixuiIiIPtPBg/fh6LgJp08/we+/38PDh1FSRyrUWNwQERFlUXKyCuPHn0KHDnvw8mU8nJxsEBjYB/b2FlJHK9Q454aIiCgLQkOj0aPHfpw//wwAMGJEHcyf3wxKJT9apcafABERkY7UaoG2bXfh1q1XsLBQYv16N3z1VSWpY9H/424pIiIiHcnlMixZ0hJffGGLK1f6sLDJYzhyQ0RElAn3779BSMgbtG5tDwBo3doerVqVhVzOK3nnNRy5ISIi+oTdu++gTp1N6NZtL0JC3mjaWdjkTSxuiIiIMpCQkILhw4+iW7e9iI5OQvXqVjAw4EdnXsfdUkREROm4e/c1PD334cqV5wCA8ePrYdasxjAw0JM4GX0KixsiIqIP7NgRjMGD/0JMTBKKFTPCpk3t0L69g9SxKJNY3BAREX3gwoUwxMQkoUmT0ti2rQNKlzaTOhLpgMUNERERACEEZLJ3E4TnzWuKChUs8e23taGvzzk2+Q1/YkREVOht2XITHTrsQUqKGgCgUOhh2LA6LGzyKf7UiIio0IqLS8KAAQHo0+cgDh16gA0brksdibIBd0sREVGhdONGJDw89uHmzZeQyQBv74YYMKCG1LEoG0g+crN8+XLY29vD0NAQLi4uuHjx4kf7L168GJUrV4aRkRHs7OwwevRoJCQk5FJaIiLK74QQ2LDhGurV24KbN1+iRAkTHDvmAW/vhtDTk/xjkbKBpD9FPz8/jBkzBt7e3ggMDETt2rXh5uaG58+fp9t/27ZtmDhxIry9vXHr1i389ttv8PPzw+TJk3M5ORER5VczZ57DgAGHER+fgtaty+Lq1b5o0aKM1LEoG0la3CxcuBCDBg1C//79Ua1aNaxatQrGxsZYv359uv3PnTuHRo0aoVevXrC3t0ebNm3Qs2fPT472EBERpfL0rAJzcwVmz26MgIBusLY2kToSZTPJipukpCRcvnwZrq6u/wsjl8PV1RXnz59Pd5mGDRvi8uXLmmLm/v37OHjwINq3b5/h8yQmJiI6OlrrRkREhYcQAkFB/9sjULVqMTx4MAiTJ3/Ba0MVUJIVN5GRkVCpVLCxsdFqt7GxQXh4eLrL9OrVCz/++CMaN24MAwMDlC9fHs2bN//obikfHx9YWFhobnZ2dtn6OoiIKO+Kjk5Er14H4Oy8GadPP9G0Fy1qJGEqymn5aubUyZMnMWfOHKxYsQKBgYHYs2cPDhw4gJ9++inDZSZNmoSoqCjN7fHjx7mYmIiIpHLlSgScnTdjx45gyGTArVsvpY5EuUSyQ8GtrKygp6eHiIgIrfaIiAiUKFEi3WWmTZuGPn364JtvvgEA1KxZE3FxcRg8eDCmTJkCuTxtraZUKqFUKrP/BRARUZ4khMCKFUEYM+YkkpJUKFPGDDt2uKNBg5JSR6NcItnIjUKhgLOzM44dO6ZpU6vVOHbsGBo0aJDuMm/fvk1TwOjpvbs6qxAi58ISEVG+8OZNArp334vhw48hKUmFTp3K48qVvixsChlJT+I3ZswYeHl5oW7duqhfvz4WL16MuLg49O/fHwDQt29flCpVCj4+PgAAd3d3LFy4EHXq1IGLiwvu3buHadOmwd3dXVPkEBFR4fXHH/ewe/ddGBjI8fPPzTBqlJPmelFUeEha3Hh6euLFixeYPn06wsPD4ejoiICAAM0k49DQUK2RmqlTp0Imk2Hq1Kl4+vQpihcvDnd3d8yePVuql0BERHmIl1d1/PffC/TsWQX16tlKHYckIhOFbH9OdHQ0LCwsEBUVBXNz8+xbcUoc4G/67v8esYA+z5tARJTTXr2Kx9SpZ+Dj0xQWFpxfWZDp8vnNa0sREVG+dP78M/TosQ+hoTGIikrC1q0dpI5EeUS+OhSciIhIrRaYP/8imjbdgdDQGJQvb4kffqgrdSzKQzhyQ0RE+UZk5Ft4eR3CwYMPAACenpWxZk0bmJtzlxT9D4sbIiLKF4KCnqNjxz14+jQWSqUeli5tiUGDavFoKEqDxQ0REeULpUu/O2ijcuWi8Pd3R61axSVORHkVixsiIsqzoqMTNbucrKyMcfhwN5Qtaw5TU4XEySgv44RiIiLKk06cCEXlyuuxceN1TVv16lYsbOiTWNwQEVGeolKpMXPmObi67kR4eByWLw+CWl2oTslGn4m7pYiIKM8IC4vF118fxPHjoQCA/v1r4NdfW0Iu56RhyjwWN0RElCccOfIQX399EM+fv4WJiQFWrnRFnz7VpY5F+RCLGyIiktz9+2/Qrt1uqFQCNWtawd/fHVWqFJM6FuVTLG6IiEhyDg6WmDChPl6+TMCiRc1hZGQgdSTKx1jcEBGRJA4duo/KlYvCwcESADBrVmOekI+yBY+WIiKiXJWcrML48afQvv0e9OixH0lJKgBgYUPZhiM3RESUa0JDo9Gjx36cP/8MAFC/fgkIwcO8KXuxuCEiolyxd+899OsXgNevE2BhocRvv7mha9dKUseiAojFDRER5aikJBUmTvwbixZdBgDUq1cCO3Z01My1IcpunHNDREQ5SgiBv/9+AgD4/ntnnDnTk4UN5SiO3BARUY4QQkAmk0Gp1Ie/vzuuXYtE584VpI5FhQCLGyIiylaJiSkYO/YULC2V+OmnxgDenceGozWUW1jcEBFRtrl37zU8PfcjMDACcrkMXl7VUaFCEaljUSHDOTdERJQt/P2D4eS0GYGBEShWzAh793ZhYUOS4MgNERF9lvj4ZIwefRKrV18FADRuXArbt3dE6dJmEiejworFDRERZZkQAq6uO3Hu3DPIZMCkSS6YObMR9PW5Y4Ckw+KGiIiyTCaTYdCgWrh79zW2bOmANm3spY5ExOKGiIh08/ZtMh49ikbVqsUAAP361UDnzhVQpIihxMmI3uG4IRERZdrNm5GoX38L2rTZhZcv4zXtLGwoL2FxQ0REmeLrex11627BjRsvkZKixsOHUVJHIkoXd0sREdFHxcYmYdiwo9i06SYAwNW1LLZsaQ8bGxOJkxGlj8UNERFl6Nq1F/Dw2Ifg4FeQy2X48cdGmDTJBXK5TOpoRBlicUNERBmaN+8igoNfoWRJU2zf3gFNm9pJHYnok1jcEBFRhpYvd4WRkT7mzGmC4sWNpY5DlCmcUExERBpXrkRg3LiTEEIAACwslFi71o2FDeUrnzVyk5CQAENDHv5HRJTfCSGwcmUQRo8+iaQkFapVK4b+/WtKHYsoS3QeuVGr1fjpp59QqlQpmJqa4v79+wCAadOm4bfffsv2gERElLOiohLh4bEPw4YdQ1KSCu7u5dG5cwWpYxFlmc7FzaxZs+Dr64uff/4ZCoVC016jRg2sW7cuW8MREVHOunQpDHXqbMKuXXdgYCDHwoXN8eefXVC0qJHU0YiyTOfiZtOmTVizZg169+4NPT09TXvt2rURHBycreGIiCjnrF9/DY0abceDB1GwtzfHmTM9MXp0XchkPMyb8jed59w8ffoUFSqkHa5Uq9VITk7OllBERJTzKlSwhEol8NVXFfHbb26wtOQcSioYdC5uqlWrhtOnT6Ns2bJa7bt27UKdOnWyLRgREWW/N28SNEVM06Z2uHChN5ydbThaQwWKzsXN9OnT4eXlhadPn0KtVmPPnj24ffs2Nm3ahP379+dERiIi+kxqtcDChf9i9ux/cP58L1Sp8u6K3nXrlpA4GVH203nOTefOnbFv3z4cPXoUJiYmmD59Om7duoV9+/ahdevWOZGRiIg+Q2TkW3Tq9DvGjTuFN28SsXnzTakjEeWoLJ3npkmTJjhy5Eh2ZyEiomx25swT9Ox5AE+exECp1MOSJS0xeHAtqWMR5SidR24cHBzw8uXLNO1v3ryBg4NDtoQiIqLPo1YL+PhcQPPmfnjyJAaVKhXBhQu98e23tTm/hgo8nUduHj58CJVKlaY9MTERT58+zZZQRET0eXx9r2Py5NMAgK+/roaVK11haqr4xFJEBUOmi5u9e/dq/n/48GFYWFho7qtUKhw7dgz29vbZGo6IiLKmb9/q2LEjGD16VEH//jU4WkOFSqaLmy5dugAAZDIZvLy8tB4zMDCAvb09FixYkK3hiIgoc1QqNX777Rr69asBhUIP+vpyHD7cjUUNFUqZLm7UajUAoFy5crh06RKsrKxyLBQREWVeeHgcevc+gOPHQxEc/AoLF7YAABY2VGjpPOfmwYMHOZGDiIiy4OjRR/j66wOIiHgLY2N91KljLXUkIsll6VDwuLg4nDp1CqGhoUhKStJ6bOTIkdkSjIiIMpaSosbMmecwe/Y/EAKoWdMK/v7umpPzERVmOhc3V65cQfv27fH27VvExcWhaNGiiIyMhLGxMaytrVncEBHlsKdPY9Cr1wH8/fcTAMCgQbWwZEkLGBkZSJyMKG/Q+Tw3o0ePhru7O16/fg0jIyP8888/ePToEZydnfHLL7/kREYiInpPfHwKrlx5DlNTA2zb1gFr1rRhYUP0Hp1HboKCgrB69WrI5XLo6ekhMTERDg4O+Pnnn+Hl5YWvvvoqJ3ISERVqQgjNBOEKFYrA398d5ctbomLFIhInI8p7dB65MTAwgFz+bjFra2uEhoYCACwsLPD48ePsTUdERHj8OBrNmvnh6NFHmra2bcuxsCHKgM4jN3Xq1MGlS5dQsWJFNGvWDNOnT0dkZCQ2b96MGjVq5ERGIqJCa9++EPTrdwivXiVg2LCjuHmzP/T0dP5eSlSo6PwbMmfOHNja2gIAZs+ejSJFimDIkCF48eIFVq9ene0BiYgKo6QkFX744QQ6dfodr14loG5dGxw61JWFDVEm6DxyU7duXc3/ra2tERAQkK2BiIgKu4cPo+DpuQ8XL4YDAEaNcsK8eU2hVGbp7B1EhU62fQUIDAxEx44ddV5u+fLlsLe3h6GhIVxcXHDx4sWP9n/z5g2GDRsGW1tbKJVKVKpUCQcPHsxqbCKiPOXx42jUqbMJFy+Gw9JSid9/74zFi1uysCHSgU7FzeHDhzF27FhMnjwZ9+/fBwAEBwejS5cuqFevnuYSDZnl5+eHMWPGwNvbG4GBgahduzbc3Nzw/PnzdPsnJSWhdevWePjwIXbt2oXbt29j7dq1KFWqlE7PS0SUV5UubQZ39/L44gtbBAX1RZcuFaWORJTvyIQQIjMdf/vtNwwaNAhFixbF69evUaxYMSxcuBAjRoyAp6cnRo0ahapVq+r05C4uLqhXrx6WLVsG4N31q+zs7DBixAhMnDgxTf9Vq1Zh/vz5CA4OhoFB1s7pEB0dDQsLC0RFRcHc3DxL60hXShzgb/ru/x6xgL5J9q2biAq0kJA3sLRUolgxIwDA27fJMDCQw8BAT+JkRHmHLp/fmR65WbJkCebNm4fIyEj4+/sjMjISK1aswLVr17Bq1SqdC5ukpCRcvnwZrq6u/wsjl8PV1RXnz59Pd5m9e/eiQYMGGDZsGGxsbFCjRg3MmTMHKpUqw+dJTExEdHS01o2IKK/w9w9GnTqb0L9/AFK/axobG7CwIfoMmS5uQkJC0L17dwDAV199BX19fcyfPx+lS5fO0hNHRkZCpVLBxsZGq93Gxgbh4eHpLnP//n3s2rULKpUKBw8exLRp07BgwQLMmjUrw+fx8fGBhYWF5mZnZ5elvERE2SkhIQVDhhyBp+d+xMQk4dWrBERHJ316QSL6pEwXN/Hx8TA2NgYAyGQyKJVKzSHhuUWtVsPa2hpr1qyBs7MzPD09MWXKFKxatSrDZSZNmoSoqCjNjScaJCKp3bnzCl98sRWrVl0FAEya5IKTJz1hYaGUOBlRwaDT9Pt169bB1PTdvJKUlBT4+vrCyspKq09mL5xpZWUFPT09REREaLVHRESgRIkS6S5ja2sLAwMD6On9b7i2atWqCA8PR1JSEhQKRZpllEollEr+wSCivGHr1pv49tsjiItLRvHiRti8uT3c3MpJHYuoQMl0cVOmTBmsXbtWc79EiRLYvHmzVh+ZTJbp4kahUMDZ2RnHjh1Dly5dALwbmTl27BiGDx+e7jKNGjXCtm3boFarNZeAuHPnDmxtbdMtbIiI8pK3b5MxdeoZxMUlo3lzO2zd2gElS5pKHYuowMl0cfPw4cNsf/IxY8bAy8sLdevWRf369bF48WLExcWhf//+AIC+ffuiVKlS8PHxAQAMGTIEy5Ytw6hRozBixAjcvXsXc+bMyXRBRUQkJWNjA/j5uePgwfuYNq0BzzZMlEMkPSuUp6cnXrx4genTpyM8PByOjo4ICAjQTDIODQ3VjNAAgJ2dHQ4fPozRo0ejVq1aKFWqFEaNGoUJEyZI9RKIiD5q48brUKkEBgyoCQCoX98W9evn7nxFosIm0+e5KSh4nhsiyg2xsUkYNuwoNm26CaVSD//954VKlYpKHYso39Ll85vn8yYiymbXrr2Ah8c+BAe/glwuw9SpX6B8eUupYxEVGixuiIiyiRACv/12DSNGHEdCQgpKljTFtm0d0KwZz69FlJtY3BARZQMhBLy8DmHz5psAgLZt7bFpU3sUL24scTKiwidLU/VDQkIwdepU9OzZU3ORy0OHDuHGjRvZGo6IKL+QyWSoWLEI9PRkmDu3CQ4c6MrChkgiOhc3p06dQs2aNXHhwgXs2bMHsbGxAICrV6/C29s72wMSEeVVQgi8fp2guT95sgsuX+6DCRNcIJfLJExGVLjpXNxMnDgRs2bNwpEjR7ROnNeyZUv8888/2RqOiCiviopKhKfnPjRv7of4+GQAgJ6eHLVrW0ucjIh0Lm6uXbuGL7/8Mk27tbU1IiMjsyUUEVFe9u+/4XBy2oSdO+/g5s2XOHv2mdSRiOg9Ohc3lpaWCAsLS9N+5coVlCpVKltCERHlRUIILF0aiIYNt+H+/SiULWuOM2d6wtW1rNTRiOg9Ohc3PXr0wIQJExAeHg6ZTAa1Wo2zZ89i7Nix6Nu3b05kJCKS3OvXCfjqqz8xatRxJCer0aVLBVy50hcuLjzbMFFeo3NxM2fOHFSpUgV2dnaIjY1FtWrV0LRpUzRs2BBTp07NiYxERJIbOvQo/vjjHhQKPSxd2hJ79nRGkSKGUscionRk+fILoaGhuH79OmJjY1GnTh1UrFgxu7PlCF5+gYiyIjQ0Gt267cXKla5wdi4hdRyiQidHL79w5swZNG7cGGXKlEGZMmWyHJKIKC97+TIe+/aFoF+/GgCAMmXMceFCb8hkPMSbKK/TebdUy5YtUa5cOUyePBk3b97MiUxERJI6e/YpHB03oX//AOzbF6JpZ2FDlD/oXNw8e/YMP/zwA06dOoUaNWrA0dER8+fPx5MnT3IiHxFRrlGrBebOvYBmzXbgyZMYVKxYBHZ2ZlLHIiId6VzcWFlZYfjw4Th79ixCQkLQvXt3bNy4Efb29mjZsmVOZCQiynHPn8ehffvdmDTpNFQqgV69quLy5T5wdORJ+Yjym8+6cGa5cuUwceJE1K5dG9OmTcOpU6eyKxcRUa45deoxevbcj7CwOBga6mPZslYYMKAGd0MR5VNZunAmAJw9exZDhw6Fra0tevXqhRo1auDAgQPZmY2IKFeEhcUhLCwOVasWxaVLvTFwYE0WNkT5mM4jN5MmTcKOHTvw7NkztG7dGkuWLEHnzp1hbMyr3xJR/iGE0BQwPXpUQVKSCl27VoSJieITSxJRXqfzyM3ff/+NcePG4enTp9i/fz969uzJwoaI8pVjxx7ByWkzwsPjNG19+1ZnYUNUQOg8cnP27NmcyEFElONUKjVmzjyHWbP+gRDAzJnnsHJla6ljEVE2y1Rxs3fvXrRr1w4GBgbYu3fvR/t26tQpW4IREWWnZ89i0avXfpw69e60Fd98UxMLFjSXNhQR5YhMFTddunRBeHg4rK2t0aVLlwz7yWQyqFSq7MpGRJQtDh9+gK+/PojIyHiYmhpg9eo26NWrqtSxiCiHZKq4UavV6f6fiCiv27nzNjw89gEAatcuDn9/d1SqVFTiVESUk3SeULxp0yYkJiamaU9KSsKmTZuyJRQRUXZp27YcKlUqgqFDHfHPP71Z2BAVAjpfFVxPTw9hYWGwttY+a+fLly9hbW2d53dL8argRAXfP/88g4uLreZQ7+joRJibKyVORUSfQ5fPb51Hbt4/N8T7njx5AgsLC11XR0SUbZKSVBg79iQaNNiGxYsva9pZ2BAVLpk+FLxOnTqQyWSQyWRo1aoV9PX/t6hKpcKDBw/Qtm3bHAlJRPQpDx9GoUeP/bhwIQwA8PRprMSJiEgqmS5uUo+SCgoKgpubG0xNTTWPKRQK2Nvbo2vXrtkekIjoU/744y769w/AmzeJsLRUYsOGtujSpaLUsYhIIpkubry9vQEA9vb28PT0hKGhYY6FIiLKjMTEFIwf/zeWLg0EALi42GLHjo6wt+cucqLCTOc5N15eXixsiChPuHnzJVasCAIA/PBDXfz9dw8WNkSUuZGbokWL4s6dO7CyskKRIkU+erXcV69eZVs4IqKPqVPHBr/+2hKlS5uhY8fyUschojwiU8XNokWLYGZmpvn/x4obIqKckpCQggkT/sbAgTVRq1ZxAMB33zlKG4qI8hydz3OT3/E8N0T50507r+DhsQ9Xr75AlSpFce1aP+jr67xnnYjyqRw9z01gYCCuXbumuf/nn3+iS5cumDx5MpKSknRPS0T0Cdu23YKz82ZcvfoCxYsbYfHiFixsiChDOv91+Pbbb3Hnzh0AwP379+Hp6QljY2Ps3LkT48ePz/aARFR4vX2bjEGDDqN37wOIjU1Gs2alERTkBTe3clJHI6I8TOfi5s6dO3B0dAQA7Ny5E82aNcO2bdvg6+uL3bt3Z3c+IiqkwsPj4OKyFevWXYNMBkyf3gBHj3qgZEnTTy9MRIVaps9zk0oIobky+NGjR9GxY0cAgJ2dHSIjI7M3HREVWsWLG8Ha2hg2NsbYurUDWrUqK3UkIsondC5u6tati1mzZsHV1RWnTp3CypUrAQAPHjyAjY1NtgckosIjLi4JenpyGBrqQ09Pjq1bOwAASpTgBH0iyjydd0stXrwYgYGBGD58OKZMmYIKFSoAAHbt2oWGDRtme0AiKhyuX3+BevW2YPToE5q2EiVMWNgQkc6y7VDwhIQE6OnpwcDAIDtWl2N4KDhR3iKEwPr11zF8+DEkJKSgZElT/PefF4oVM5I6GhHlIbp8fuu8WyrV5cuXcevWLQBAtWrV4OTklNVVEVEhFROThCFDjmDr1nd/S9zc7LF5c3sWNkT0WXQubp4/fw5PT0+cOnUKlpaWAIA3b96gRYsW2LFjB4oXL57dGYmoALp69Tk8PPbhzp3X0NOTYdasxhg/vj7kcp4BnYg+j85zbkaMGIHY2FjcuHEDr169wqtXr3D9+nVER0dj5MiROZGRiAqYxMQUtG+/B3fuvEbp0mY4daoHJk50YWFDRNlC55GbgIAAHD16FFWrVtW0VatWDcuXL0ebNm2yNRwRFUxKpT5WrnTF2rX/wde3HXdDEVG20rm4UavV6U4aNjAw0Jz/hojoQ5cvh+P160S4ur47X02nThXg7l6eF+Ilomyn826pli1bYtSoUXj27Jmm7enTpxg9ejRatWqVreGIKP8TQuDXXwPRsOF2eHruw+PH0ZrHWNgQUU7QubhZtmwZoqOjYW9vj/Lly6N8+fIoV64coqOj8euvv+ZERiLKp16/TkDXrnsxcuRxJCWp0LRpaZiaKqSORUQFnM67pezs7BAYGIhjx45pDgWvWrUqXF1dsz0cEeVfFy6EoUePfXj4MBoKhR5++aUZhg+vw9EaIspxOhU3fn5+2Lt3L5KSktCqVSuMGDEip3IRUT4lhMCiRZcxYcLfSElRw8HBAv7+7nB2LiF1NCIqJDJd3KxcuRLDhg1DxYoVYWRkhD179iAkJATz58/PyXxElM/IZDIEB79CSooa3btXwtq1brCwUEodi4gKkUzPuVm2bBm8vb1x+/ZtBAUFYePGjVixYkVOZiOifESt/t+VXJYsaYEtW9rDz8+dhQ0R5bpMFzf379+Hl5eX5n6vXr2QkpKCsLCwHAlGRPmDWi0wb94FdOy4R1PgGBkZoHfvapxfQ0SSyPRuqcTERJiY/O9ikHK5HAqFAvHx8TkSjIjyvhcv3qJv34MICHgIAPjzz3v48suK0oYiokJPpwnF06ZNg7GxseZ+UlISZs+eDQsLC03bwoULsy8dEeVZf//9GD17HsCzZ7EwNNTHsmWt0KVLBaljERFlvrhp2rQpbt++rdXWsGFD3L9/X3OfQ9BEBZ9KpYaPzwV4e5+DWi1QtWpR+Pu7o0YNXjSXiPKGTBc3J0+ezMEYRJRfDB16FGvW/AcA6NevOpYtawUTE56Yj4jyDp3PUJwTli9fDnt7exgaGsLFxQUXL17M1HI7duyATCZDly5dcjYgEWkMGeKIokUNsXFjO2zY0I6FDRHlOZIXN35+fhgzZgy8vb0RGBiI2rVrw83NDc+fP//ocg8fPsTYsWPRpEmTXEpKVDipVGqcP/+/a8k5Olrj0aPB6Nu3uoSpiIgyJnlxs3DhQgwaNAj9+/dHtWrVsGrVKhgbG2P9+vUZLqNSqdC7d2/MnDkTDg4OuZiWqHB59iwWrVr5o1mzHbh06X+nfeD1oYgoL5O0uElKSsLly5e1rksll8vh6uqK8+fPZ7jcjz/+CGtrawwcODA3YhIVSocPP4Cj40acOvUESqUenj2LkzoSEVGm6HzhzOwUGRkJlUoFGxsbrXYbGxsEBwenu8yZM2fw22+/ISgoKFPPkZiYiMTERM396OjoLOclKgxSUtSYNu0M5s59N/etdu3i8Pd3R6VKRSVORkSUOVkauTl9+jS+/vprNGjQAE+fPgUAbN68GWfOnMnWcB+KiYlBnz59sHbtWlhZWWVqGR8fH1hYWGhudnZ2OZqRKD97/DgazZv7aQqboUMd8c8/vVnYEFG+onNxs3v3bri5ucHIyAhXrlzRjIpERUVhzpw5Oq3LysoKenp6iIiI0GqPiIhAiRJpryAcEhKChw8fwt3dHfr6+tDX18emTZuwd+9e6OvrIyQkJM0ykyZNQlRUlOb2+PFjnTISFSZ79tzF2bNPYW6ugL+/O5Yvd4WhoaQDvEREOtO5uJk1axZWrVqFtWvXwsDAQNPeqFEjBAYG6rQuhUIBZ2dnHDt2TNOmVqtx7NgxNGjQIE3/KlWq4Nq1awgKCtLcOnXqhBYtWiAoKCjdURmlUglzc3OtGxGlb8QIJ4wfXw+BgX3RvXtlqeMQEWWJzl/Jbt++jaZNm6Zpt7CwwJs3b3QOMGbMGHh5eaFu3bqoX78+Fi9ejLi4OPTv3x8A0LdvX5QqVQo+Pj4wNDREjRo1tJa3tLQEgDTtRPRpjx5FYdq0s1ixwhWmpgrI5TLMm9dM6lhERJ9F5+KmRIkSuHfvHuzt7bXaz5w5k6XDsj09PfHixQtMnz4d4eHhcHR0REBAgGaScWhoKORyyY9YJypw/vzzHvr1O4Q3bxJhamqAFStaSx2JiChb6FzcDBo0CKNGjcL69eshk8nw7NkznD9/HmPHjsW0adOyFGL48OEYPnx4uo996rIPvr6+WXpOosIqKUmF8eNPYcmSd7uR69cvgfHj60uciogo++hc3EycOBFqtRqtWrXC27dv0bRpUyiVSowdOxYjRozIiYxElE3u338DT899+Pffd5P4f/ihLubMaQKFQk/iZERE2UcmhBBZWTApKQn37t1DbGwsqlWrBlNT0+zOliOio6NhYWGBqKio7J1cnBIH+P//NvCIBfRNsm/dRNng5MlQdO78B6KjkzTXhurYsbzUsYiIMkWXz+8sH+OpUChQrVq1rC5ORLmscuWiMDTUR82axbF9ewfY2fHIQSIqmHQublq0aAGZTJbh48ePH/+sQESUfSIj38LKyhgAYGtrilOnPFG+vCUMDLgbiogKLp0PQ3J0dETt2rU1t2rVqiEpKQmBgYGoWbNmTmQkoizYvv0WHBzWYdeu25q2KlWKsbAhogJP55GbRYsWpds+Y8YMxMbGfnYgIvo88fHJGDXqBNau/Q8AsGnTTXTrxhPyEVHhkW0nkPn666+xfv367FodEWVBcPBLuLhsxdq1/0EmA6ZN+wJ79nSWOhYRUa7KtovGnD9/HoaGhtm1OiLS0aZNNzBkyBG8fZsCGxtjbNnSAa6uZaWORUSU63Qubr766iut+0IIhIWF4d9//83ySfyI6PMEBkbAy+sQAKBlyzLYurUDSpTg6QiIqHDSubixsLDQui+Xy1G5cmX8+OOPaNOmTbYFI6LMc3KywQ8/1IWFhRKTJ7tAT4+XLCGiwkun4kalUqF///6oWbMmihQpklOZiOgThBDYtOkGWrUqi9KlzQAAv/zSXNpQRER5hE5f7/T09NCmTZssXf2biLJHTEwS+vQ5iH79AtCz536kpKiljkRElKfoPHZdo0YN3L9/PyeyENEnXL36HHXrbsbWrbegpydDhw4OkMszPqkmEVFhpHNxM2vWLIwdOxb79+9HWFgYoqOjtW5ElP2EEFi9+ipcXLbizp3XKF3aDKdO9cDEiS4sboiIPpDpOTc//vgjfvjhB7Rv3x4A0KlTJ63LMAghIJPJoFKpsj8lUSEWE5OEb745DH//d2ca7tjRAb6+7VCsmJHEyYiI8qZMFzczZ87Ed999hxMnTuRkHiL6gJ6eDDdvvoS+vhxz5zbBmDF1P3p9NyKiwi7TxY0QAgDQrFmzHAtDRO8IISAEIJfLYGxsAH9/d0RFJeKLL0pKHY2IKM/Tac4Nvy0S5bw3bxLQrdtezJt3UdNWtWoxFjZERJmk03luKlWq9MkC59WrV58ViKgwu3gxDJ6e+/DwYTQOHXqAAQNqwMaGZxomItKFTsXNzJkz05yhmIg+nxACixdfxoQJfyM5WQ0HBwv4+bmzsCEiygKdipsePXrA2to6p7IQFUqvXsWjX78A7NsXAgDo1q0S1q1zg4WFUuJkRET5U6aLG863Icp+SUkqfPHFNty9+xpKpR4WLWqB776rzd83IqLPkOkJxalHSxFR9lEo9PD9906oWLEI/vmnN4YMcWRhQ0T0mTI9cqNW8/o1RNkhMvItnj9/i2rVrAAAQ4Y4ol+/GjA2NpA4GRFRwaDz5ReIKOtOn36C2rU3wd39d0RFJQJ4t8uXhQ0RUfZhcUOUC9Rqgdmz/0Hz5n549iwWCoUeXrx4K3UsIqICSaejpYhIdxERcejT5yCOHHkEAPDyqo7ly1vBxEQhcTIiooKJxQ1RDjp+PBS9ex9AeHgcjI31sWKFK7y8akgdi4ioQGNxQ5SDFi36F+HhcahevRj8/d01k4iJiCjncM4NUQ7asKEtxo6ti4sXv2ZhQ0SUS1jcEGWjv/56iLFjT2ruW1kZY/785jwaiogoF3G3FFE2SElRw9v7LHx8LkAIoGHDkvjqq0pSxyIiKpRY3BB9pidPYtCr1wGcPv0EAPDdd7XRrl05iVMRERVeLG6IPsPBg/fRt+8hvHwZDzMzBdatawMPjypSxyIiKtRY3BBl0Zw5/2DKlDMAAGdnG/j5uaN8eUtpQxEREScUE2WVs7MNZDJgxIg6OHu2JwsbIqI8giM3RDp4/jwO1tYmAAA3t3K4caM/qlYtJnEqIiJ6H0duiDIhKUmF0aNPoHLl9bh//42mnYUNEVHew+KG6BMePHiDxo23Y/Hiy3jzJhGHDj2QOhIREX0Ed0sRfcTu3XcwcOBhREUlomhRQ/j6toO7e3mpYxER0UewuCFKR0JCCsaOPYnly4MAvDsp3/btHVGmjLm0wYiI6JO4W4ooHUuXBmoKmwkT6uPkSU8WNkRE+QRHbojSMWqUE06cCMXIkU5o185B6jhERKQDjtwQAYiPT8Yvv1xCSooaAKBU6uPQoW4sbIiI8iGO3FChFxz8Eh4e+3DtWiTevEnErFmNpY5ERESfgcUNFWqbN9/AkCFHEReXDBsbYzRvbid1JCIi+kwsbqhQiotLwogRx7Fhw3UAQMuWZbB1aweUKGEicTIiIvpcLG6o0Ll16yW6dduLmzdfQi6Xwdu7AaZM+QJ6epyCRkRUELC4oUJHrRZ48CAKtrYm2LatA5o3LyN1JCIiykYsbqhQUKnUmpGZ6tWt8PvvnVGnjrXmIphERFRwcByeCryrV5+jVq2NOHPmiabNza0cCxsiogKKxQ0VWEIIrF59FS4uW3Hz5kuMG3cKQgipYxERUQ7jbikqkKKjEzF48F/w87sNAGjfvhw2bmwHmUwmcTIiIsppLG6owAkMjICn5z7cu/cG+vpy+Pg0wZgxdSGXs7AhIioMWNxQgXL9+gs0aLANSUkqlCljhh073NGgQUmpYxERUS5icUMFSvXqVujY0QEpKWps2NAWRYsaSR2JiIhyWZ6YULx8+XLY29vD0NAQLi4uuHjxYoZ9165diyZNmqBIkSIoUqQIXF1dP9qfCr5//w1HVFQiAEAmk2HLlvb4448uLGyIiAopyYsbPz8/jBkzBt7e3ggMDETt2rXh5uaG58+fp9v/5MmT6NmzJ06cOIHz58/Dzs4Obdq0wdOnT3M5OUlNCIFFi/5Fw4bbMHjwX5ojoYyMDDhxmIioEJMJiY+NdXFxQb169bBs2TIAgFqthp2dHUaMGIGJEyd+cnmVSoUiRYpg2bJl6Nu37yf7R0dHw8LCAlFRUTA3N//s/BopcYC/6bv/e8QC+jyHSk569Soe/fsHYO/eEABAt26VsGVLeyiV3NNKRFQQ6fL5LenITVJSEi5fvgxXV1dNm1wuh6urK86fP5+pdbx9+xbJyckoWrRoTsWkPOb8+WdwdNyEvXtDoFDoYfnyVvD3d2dhQ0REACSeUBwZGQmVSgUbGxutdhsbGwQHB2dqHRMmTEDJkiW1CqT3JSYmIjExUXM/Ojo664FJUmq1wC+/XMLkyaehUglUqGAJf3931Klj8+mFiYio0JB8zs3nmDt3Lnbs2IHff/8dhoaG6fbx8fGBhYWF5mZnZ5fLKSm7vHmTgCVLAqFSCfTsWQWBgX1Z2BARURqSFjdWVlbQ09NDRESEVntERARKlCjx0WV/+eUXzJ07F3/99Rdq1aqVYb9JkyYhKipKc3v8+HG2ZKfcV7SoEbZv74A1a9pg69YOMDNTSB2JiIjyIEmLG4VCAWdnZxw7dkzTplarcezYMTRo0CDD5X7++Wf89NNPCAgIQN26dT/6HEqlEubm5lo3yh/UaoHZs//Bli03NW1Nm9ph0KBaPBqKiIgyJPkMzDFjxsDLywt169ZF/fr1sXjxYsTFxaF///4AgL59+6JUqVLw8fEBAMybNw/Tp0/Htm3bYG9vj/DwcACAqakpTE1NJXsdlL0iIuLQp89BHDnyCMbG+mjRwg6lSplJHYuIiPIByYsbT09PvHjxAtOnT0d4eDgcHR0REBCgmWQcGhoKufx/A0wrV65EUlISunXrprUeb29vzJgxIzejUw45cSIUvXodQHh4HIyM9LFsWSuULMnClYiIMkfy89zkNp7nJu9SqdSYNesf/PjjeajVAtWrF4O/vzuqVbOSOhoREUlMl89vyUduiAAgJUWNtm134dixUADAwIE1sXRpSxgbG0icjIiI8pt8fSg4FRz6+nLUq1cCJiYG2LKlPdatc2NhQ0REWcKRG5JMSooar18noHhxYwDAjz82wjff1EL58pbSBiMionyNIzckiSdPYtCihR86dNiDpCQVAMDAQI+FDRERfTYWN5TrDh68D0fHTThz5imCg1/h+vVIqSMREVEBwuKGck1ysgrjx59Chw578PJlPJycbBAY2AdOTryEAhERZR/OuaFc8ehRFHr02I9//gkDAIwYUQfz5zfjlbyJiCjb8ZOFcsU33/yFf/4Jg4WFEuvXu+GrrypJHYmIiAoo7paiXLFypStcXcviypU+LGyIiChHsbihHPHgwRusW/ef5n6FCkVw5Eh3lCtnKV0oIiIqFLhbirLd7t13MHDgYURHJ8Le3gKurmWljkRERIUIixvKNgkJKRg79iSWLw8CADRoUBIVK1pKmomIiAofFjeULe7dew0Pj324cuU5AGD8+HqYNasxDAz0JE5GRESFDYsb+mw7d97GwIGHEROThGLFjLBpUzu0b+8gdSwiIiqkWNzQZ4uNTUJMTBKaNCmNbds6oHRpM6kjERFRIcbihrIkJUUNff13B9v161cDpqYKfPllRU0bERGRVPhJRDrbvPkGatXyxcuX8QAAmUyG7t0rs7AhIqI8gZ9GlGlxcUkYMCAAffsewq1br7B0aaDUkYiIiNLgbinKlBs3IuHhsQ83b76ETAZ4ezfE1KlfSB2LiIgoDRY39FFCCPj6XsewYccQH5+CEiVMsG1bB7RoUUbqaEREROlicUMftWJFEIYPPwYAaN26LDZvbg8bGxOJUxEREWWMc27oo3r3rooKFSwxe3ZjBAR0Y2FDRER5HkduSIsQAkePPoKra1nIZDJYWhri2rV+MDTkW4WIiPIHjtyQRnR0Inr1OoA2bXZh7dr/XdGbhQ0REeUn/NQiAMCVKxHw8NiHe/feQF9fjvj4FKkjERFliVqtRlJSktQxKAsUCgXk8s8fd2FxU8gJIbBiRRDGjDmJpCQVypQxw44d7mjQoKTU0YiIdJaUlIQHDx5ArVZLHYWyQC6Xo1y5clAoFJ+1HhY3hdibNwn45pvD2L37LgCgU6fy2LChLYoWNZI4GRGR7oQQCAsLg56eHuzs7LJlBIByj1qtxrNnzxAWFoYyZcpAJpNleV0sbgqxa9ci8fvv92BgIMfPPzfDqFFOn/VmIiKSUkpKCt6+fYuSJUvC2NhY6jiUBcWLF8ezZ8+QkpICAwODLK+HxU0h1qRJaSxb1gp169qgXj1bqeMQEX0WlUoFAJ+9S4Okk/qzU6lUn1XccMyuEHn1Kh69eu3H7duvNG1DhjiysCGiAoUj0PlXdv3sOHJTSJw//ww9euxDaGgM7t17gwsXevMPABERFUgcuSng1GqB+fMvomnTHQgNjUH58pZYtao1CxsiojyiX79+kMlkkMlkMDAwQLly5TB+/HgkJCRo9du/fz+aNWsGMzMzGBsbo169evD19U13nbt370bz5s1hYWEBU1NT1KpVCz/++CNevXqVbv+ChsVNARYZ+Rbu7nswfvzfSElRw9OzMgID+8DJyUbqaERE9J62bdsiLCwM9+/fx6JFi7B69Wp4e3trHv/111/RuXNnNGrUCBcuXMB///2HHj164LvvvsPYsWO11jVlyhR4enqiXr16OHToEK5fv44FCxbg6tWr2Lx5c26/NElwt1QBde/eazRv7oenT2NhaKiPJUtaYNCgWhyxISLKg5RKJUqUKAEAsLOzg6urK44cOYJ58+bh8ePH+OGHH/D9999jzpw5mmV++OEHKBQKjBw5Et27d4eLiwsuXryIOXPmYPHixRg1apSmr729PVq3bo03b97k9kuTBIubAqpsWXOULWsOU1MF/P3dUatWcakjERHlLiEA1VtpnlvPGMjil8nr16/j3LlzKFu2LABg165dSE5OTjNCAwDffvstJk+ejO3bt8PFxQVbt26Fqakphg4dmu66LS0ts5Qpv2FxU4C8ePEWFhZKKBR6MDDQw65dnWBmpoCpKQ+LJKJCSPUW8DeV5rk9YgF9k0x3379/P0xNTZGSkoLExETI5XIsW7YMAHDnzh1YWFjA1jbtka0KhQIODg64c+cOAODu3btwcHD4rMOoCwLOuSkgTpwIRa1aGzF58mlNm62tKQsbIqJ8oEWLFggKCsKFCxfg5eWF/v37o2vXrjqvRwiRA+nyH47c5HMqlRqzZv2DH388D7VaICDgAX78sRGMjQt31U5EBD3jdyMoUj23DkxMTFChQgUAwPr161G7dm389ttvGDhwICpVqoSoqCg8e/YMJUtqX/cvKSkJISEhaNGiBQCgUqVKOHPmDJKTkwv16A1HbvKxsLBYtGmzCzNmnINaLTBgQA1cvPg1CxsiIuDdnBd9E2lun3Hwhlwux+TJkzF16lTEx8eja9euMDAwwIIFC9L0XbVqFeLi4tCzZ08AQK9evRAbG4sVK1aku+7CMqGYxU0+deTIQzg6bsLx46EwMTHApk3t8NtvbVnYEBEVAN27d4eenh6WL1+OMmXK4Oeff8bixYsxZcoUBAcHIyQkBAsXLsT48ePxww8/wMXFBQDg4uKiaRs/fjzOnz+PR48e4dixY+jevTs2btwo8SvLHdwtlQ+9eZOA7t33ISoqETVrWsHf3x1VqhSTOhYREWUTfX19DB8+HD///DOGDBmC77//Hg4ODvjll1+wZMkSqFQqVK9eHStXrkT//v21lp03bx6cnZ2xfPlyrFq1Cmq1GuXLl0e3bt3g5eUl0SvKXTJRyGYfRUdHw8LCAlFRUTA3N8++FafE/W9Wvo6z5LNix45gnDgRisWLW8DIiKM1REQJCQl48OABypUrB0NDQ6njUBZ87Geoy+c3R27yiUOH7sPQUB8tWpQBAPToUQU9elSROBUREVHewzk3eVxysgoTJpxC+/Z70LPnfkRExEkdiYiIKE/jyE0eFhoajR499uP8+WcAgG7dKsHCQilxKiIioryNxU0etXfvPfTrF4DXrxNgYaHEb7+5oWvXSlLHIiIiyvNY3OQxKpUa48adwqJFlwEA9eqVwI4dHeHgYCltMCIionyCc27yGLlchufP313o7fvvnXHmTE8WNkRERDrgyE0ekZKihr6+HDKZDCtXtkbv3lXRrp2D1LGIiIjyHY7cSCwxMQUjRhxD165/ai54ZmamYGFDRESURRy5kdC9e6/h6bkfgYERAIAzZ56iSZPSEqciIiLK3zhyIxE/v2A4OW1GYGAEihUzwv79X7KwISKiPGvGjBlwdHSUOkamsLjJZfHxyfjuuyPo0WM/YmKS0LhxKQQF9UWHDuWljkZERBIKDw/HqFGjUKFCBRgaGsLGxgaNGjXCypUr8fbtW6njYezYsTh27JjUMTKFu6VyWY8e+7F3bwhkMmDSJBfMnNkI+vqsMYmICrP79++jUaNGsLS0xJw5c1CzZk0olUpcu3YNa9asQalSpdCpUydJM5qamsLU1FTSDJnFT9VcNnnyFyhVyhQBAd0we3YTFjZERIShQ4dCX18f//77Lzw8PFC1alU4ODigc+fOOHDgANzd3fHw4UPIZDIEBQVplnvz5g1kMhlOnjypabt+/TratWsHU1NT2NjYoE+fPoiMjNQ8vmvXLtSsWRNGRkYoVqwYXF1dERf37tI+J0+eRP369WFiYgJLS0s0atQIjx49ApB2t1S/fv3QpUsX/PLLL7C1tUWxYsUwbNgwJCcna/qEhYWhQ4cOMDIyQrly5bBt2zbY29tj8eLFObIdU3HkJoe9fZuMS5fC0ayZHQDAxcUWISHfQKnkpiciyg1xcUkZPqanJ4ehoX6m+srlMhgZGXyyr4mJQqd8L1++xF9//YU5c+bAxMQk3T4ymSxT63rz5g1atmyJb775BosWLUJ8fDwmTJgADw8PHD9+HGFhYejZsyd+/vlnfPnll4iJicHp06chhEBKSgq6dOmCQYMGYfv27UhKSsLFixc/+twnTpyAra0tTpw4gXv37sHT0xOOjo4YNGgQAKBv376IjIzEyZMnYWBggDFjxuD58+c6bZ+s4CdsDrp5MxIeHvsQEhKFCxd6o1at4gDAwoaIKBeZmi7N8LH27cvhwIGumvvW1ivw9m1Kun2bNSuNkyd7aO7b269FZGR8mn5CjNUp37179yCEQOXKlbXarayskJCQAAAYNmwYhgwZ8sl1LVu2DHXq1MGcOXM0bevXr4ednR3u3LmD2NhYpKSk4KuvvkLZsmUBADVr1gQAvHr1ClFRUejYsSPKl383D7Rq1aoffb4iRYpg2bJl0NPTQ5UqVdChQwccO3YMgwYNQnBwMI4ePYpLly6hbt26AIB169ahYsWKmdwyWZcn9oksX74c9vb2MDQ0hIuLCy5evPjR/jt37kSVKlVgaGiImjVr4uDBg7mUNHOEENiw4Rrq1t2CGzdewtJSiejoRKljERFRPnLx4kUEBQWhevXqSEzM3GfI1atXceLECc38GFNTU1SpUgUAEBISgtq1a6NVq1aoWbMmunfvjrVr1+L169cAgKJFi6Jfv35wc3ODu7s7lixZgrCwsI8+X/Xq1aGnp6e5b2trqxmZuX37NvT19eHk5KR5vEKFCihSpIhO2yErJB9C8PPzw5gxY7Bq1Sq4uLhg8eLFcHNzw+3bt2FtbZ2m/7lz59CzZ0/4+PigY8eO2LZtG7p06YLAwEDUqFFDglegLTZBgaH9j2Pz1jsAgNaty2Lz5vawsUl/qJGIiHJWbOzIDB/T09P+jv/8+dAM+8rl2rtnHj4c9HnB/l+FChUgk8lw+/ZtrXYHh3cnczUyMvr/53+XNfWErwC05rcAQGxsLNzd3TFv3rw0z2Nraws9PT0cOXIE586dw19//YVff/0VU6ZMwYULF1CuXDls2LABI0eOREBAAPz8/DB16lQcOXIEX3zxRbrZDQwMtO7LZDKo1Wodt0D2k3zkZuHChRg0aBD69++PatWqYdWqVTA2Nsb69evT7b9kyRK0bdsW48aNQ9WqVfHTTz/ByckJy5Yty+Xkaf0Xaot600Zi89Y7kMtlmDWrMQICurGwISKSkImJIsPb+/NtPtX3/fk2H+urq2LFiqF169ZYtmyZZmJveooXfze14f3RlPcnFwOAk5MTbty4AXt7e1SoUEHrljqfRyaToVGjRpg5cyauXLkChUKB33//XbOOOnXqYNKkSTh37hxq1KiBbdu26fyaAKBy5cpISUnBlStXNG337t3TjBTlJEmLm6SkJFy+fBmurq6aNrlcDldXV5w/fz7dZc6fP6/VHwDc3Nwy7J+YmIjo6GitW07583J1BD+zQcmSJjhxwgNTpnyRptInIiL60IoVK5CSkoK6devCz88Pt27dwu3bt7FlyxYEBwdDT08PRkZG+OKLLzB37lzcunULp06dwtSpU7XWM2zYMLx69Qo9e/bEpUuXEBISgsOHD6N///5QqVS4cOEC5syZg3///RehoaHYs2cPXrx4gapVq+LBgweYNGkSzp8/j0ePHuGvv/7C3bt3PznvJiNVqlSBq6srBg8ejIsXL+LKlSsYPHgwjIyMMj1BOqsk3S0VGRkJlUoFGxsbrXYbGxsEBwenu0x4eHi6/cPDw9Pt7+Pjg5kzZ2ZP4E+Y3PkYklL0MHLFFhS3tcqV5yQiovyvfPnyuHLlCubMmYNJkybhyZMnUCqVqFatGsaOHYuhQ9/tLlu/fj0GDhwIZ2dnVK5cGT///DPatGmjWU/JkiVx9uxZTJgwAW3atEFiYiLKli2Ltm3bQi6Xw9zcHH///TcWL16M6OholC1bFgsWLEC7du0QERGB4OBgbNy4ES9fvoStrS2GDRuGb7/9Nsuva9OmTRg4cCCaNm2KEiVKwMfHBzdu3IChoeFnb7OPkYn3d97lsmfPnqFUqVI4d+4cGjRooGkfP348Tp06hQsXLqRZRqFQYOPGjejZs6embcWKFZg5cyYiIiLS9E9MTNSaiBUdHQ07OztERUXB3Nw8+16MEIDq/88gqWcM5HBVSkRE2hISEvDgwQOUK1cuxz88KWuePHkCOzs7HD16FK1atUrz+Md+htHR0bCwsMjU57ekIzdWVlbQ09NLU5RERESgRIkS6S5TokQJnforlUoolcrsCfwxMhmgz7k1REREqY4fP47Y2FjUrFkTYWFhGD9+POzt7dG0adMcfV5J59woFAo4OztrXatCrVbj2LFjWiM572vQoEGaa1scOXIkw/5EREQkjeTkZEyePBnVq1fHl19+ieLFi2tO6JeTJD8UfMyYMfDy8kLdunVRv359LF68GHFxcejfvz+Ad2c3LFWqFHx8fAAAo0aNQrNmzbBgwQJ06NABO3bswL///os1a9ZI+TKIiIjoA25ubnBzc8v155W8uPH09MSLFy8wffp0hIeHw9HREQEBAZpJw6GhoZpj+wGgYcOG2LZtG6ZOnYrJkyejYsWK+OOPP/LEOW6IiIhIepJOKJaCLhOSiIgo/+CE4vwvuyYUS34SPyIiouxUyL6zFyjZ9bNjcUNERAVC6jWOkpIyvrI35W2pP7v3r1eVFZLPuSEiIsoO+vr6MDY2xosXL2BgYKA1X5PyPrVajRcvXsDY2Bj6+p9XnrC4ISKiAkEmk8HW1hYPHjzAo0ePpI5DWSCXy1GmTJnPvjwDixsiIiowFAoFKlasyF1T+ZRCociWETcWN0REVKDI5XIeLVXIcYckERERFSgsboiIiKhAYXFDREREBUqhm3OTeoKg6OhoiZMQERFRZqV+bmfmRH+FrriJiYkBANjZ2UmchIiIiHQVExMDCwuLj/YpdNeWUqvVePbsGczMzD77OPoPRUdHw87ODo8fP+Z1q3IQt3Pu4HbOHdzOuYfbOnfk1HYWQiAmJgYlS5b85OHihW7kRi6Xo3Tp0jn6HObm5vzFyQXczrmD2zl3cDvnHm7r3JET2/lTIzapOKGYiIiIChQWN0RERFSgsLjJRkqlEt7e3lAqlVJHKdC4nXMHt3Pu4HbOPdzWuSMvbOdCN6GYiIiICjaO3BAREVGBwuKGiIiIChQWN0RERFSgsLghIiKiAoXFjY6WL18Oe3t7GBoawsXFBRcvXvxo/507d6JKlSowNDREzZo1cfDgwVxKmr/psp3Xrl2LJk2aoEiRIihSpAhcXV0/+XOhd3R9P6fasWMHZDIZunTpkrMBCwhdt/ObN28wbNgw2NraQqlUolKlSvzbkQm6bufFixejcuXKMDIygp2dHUaPHo2EhIRcSps//f3333B3d0fJkiUhk8nwxx9/fHKZkydPwsnJCUqlEhUqVICvr2+O54SgTNuxY4dQKBRi/fr14saNG2LQoEHC0tJSREREpNv/7NmzQk9PT/z888/i5s2bYurUqcLAwEBcu3Ytl5PnL7pu5169eonly5eLK1euiFu3bol+/foJCwsL8eTJk1xOnr/oup1TPXjwQJQqVUo0adJEdO7cOXfC5mO6bufExERRt25d0b59e3HmzBnx4MEDcfLkSREUFJTLyfMXXbfz1q1bhVKpFFu3bhUPHjwQhw8fFra2tmL06NG5nDx/OXjwoJgyZYrYs2ePACB+//33j/a/f/++MDY2FmPGjBE3b94Uv/76q9DT0xMBAQE5mpPFjQ7q168vhg0bprmvUqlEyZIlhY+PT7r9PTw8RIcOHbTaXFxcxLfffpujOfM7Xbfzh1JSUoSZmZnYuHFjTkUsELKynVNSUkTDhg3FunXrhJeXF4ubTNB1O69cuVI4ODiIpKSk3IpYIOi6nYcNGyZatmyp1TZmzBjRqFGjHM1ZkGSmuBk/fryoXr26Vpunp6dwc3PLwWRCcLdUJiUlJeHy5ctwdXXVtMnlcri6uuL8+fPpLnP+/Hmt/gDg5uaWYX/K2nb+0Nu3b5GcnIyiRYvmVMx8L6vb+ccff4S1tTUGDhyYGzHzvaxs571796JBgwYYNmwYbGxsUKNGDcyZMwcqlSq3Yuc7WdnODRs2xOXLlzW7ru7fv4+DBw+iffv2uZK5sJDqc7DQXTgzqyIjI6FSqWBjY6PVbmNjg+Dg4HSXCQ8PT7d/eHh4juXM77KynT80YcIElCxZMs0vFP1PVrbzmTNn8NtvvyEoKCgXEhYMWdnO9+/fx/Hjx9G7d28cPHgQ9+7dw9ChQ5GcnAxvb+/ciJ3vZGU79+rVC5GRkWjcuDGEEEhJScF3332HyZMn50bkQiOjz8Ho6GjEx8fDyMgoR56XIzdUoMydOxc7duzA77//DkNDQ6njFBgxMTHo06cP1q5dCysrK6njFGhqtRrW1tZYs2YNnJ2d4enpiSlTpmDVqlVSRytQTp48iTlz5mDFihUIDAzEnj17cODAAfz0009SR6NswJGbTLKysoKenh4iIiK02iMiIlCiRIl0lylRooRO/Slr2znVL7/8grlz5+Lo0aOoVatWTsbM93TdziEhIXj48CHc3d01bWq1GgCgr6+P27dvo3z58jkbOh/KyvvZ1tYWBgYG0NPT07RVrVoV4eHhSEpKgkKhyNHM+VFWtvO0adPQp08ffPPNNwCAmjVrIi4uDoMHD8aUKVMgl/O7f3bI6HPQ3Nw8x0ZtAI7cZJpCoYCzszOOHTumaVOr1Th27BgaNGiQ7jINGjTQ6g8AR44cybA/ZW07A8DPP/+Mn376CQEBAahbt25uRM3XdN3OVapUwbVr1xAUFKS5derUCS1atEBQUBDs7OxyM36+kZX3c6NGjXDv3j1N8QgAd+7cga2tLQubDGRlO799+zZNAZNaUApecjHbSPY5mKPTlQuYHTt2CKVSKXx9fcXNmzfF4MGDhaWlpQgPDxdCCNGnTx8xceJETf+zZ88KfX198csvv4hbt24Jb29vHgqeCbpu57lz5wqFQiF27dolwsLCNLeYmBipXkK+oOt2/hCPlsocXbdzaGioMDMzE8OHDxe3b98W+/fvF9bW1mLWrFlSvYR8Qdft7O3tLczMzMT27dvF/fv3xV9//SXKly8vPDw8pHoJ+UJMTIy4cuWKuHLligAgFi5cKK5cuSIePXokhBBi4sSJok+fPpr+qYeCjxs3Tty6dUssX76ch4LnRb/++qsoU6aMUCgUon79+uKff/7RPNasWTPh5eWl1d/f319UqlRJKBQKUb16dXHgwIFcTpw/6bKdy5YtKwCkuXl7e+d+8HxG1/fz+1jcZJ6u2/ncuXPCxcVFKJVK4eDgIGbPni1SUlJyOXX+o8t2Tk5OFjNmzBDly5cXhoaGws7OTgwdOlS8fv0694PnIydOnEj3723qtvXy8hLNmjVLs4yjo6NQKBTCwcFBbNiwIcdzyoTg+BsREREVHJxzQ0RERAUKixsiIiIqUFjcEBERUYHC4oaIiIgKFBY3REREVKCwuCEiIqIChcUNERERFSgsbohIi6+vLywtLaWOkWUymQx//PHHR/v069cPXbp0yZU8RJT7WNwQFUD9+vWDTCZLc7t3757U0eDr66vJI5fLUbp0afTv3x/Pnz/PlvWHhYWhXbt2AICHDx9CJpMhKChIq8+SJUvg6+ubLc+XkRkzZmhep56eHuzs7DB48GC8evVKp/WwECPSHa8KTlRAtW3bFhs2bNBqK168uERptJmbm+P27dtQq9W4evUq+vfvj2fPnuHw4cOfve5PXT0eACwsLD77eTKjevXqOHr0KFQqFW7duoUBAwYgKioKfn5+ufL8RIUVR26ICiilUokSJUpo3fT09LBw4ULUrFkTJiYmsLOzw9ChQxEbG5vheq5evYoWLVrAzMwM5ubmcHZ2xr///qt5/MyZM2jSpAmMjIxgZ2eHkSNHIi4u7qPZZDIZSpQogZIlS6Jdu3YYOXIkjh49ivj4eKjVavz4448oXbo0lEolHB0dERAQoFk2KSkJw4cPh62tLQwNDVG2bFn4+PhorTt1t1S5cuUAAHXq1IFMJkPz5s0BaI+GrFmzBiVLltS6CjcAdO7cGQMGDNDc//PPP+Hk5ARDQ0M4ODhg5syZSElJ+ejr1NfXR4kSJVCqVCm4urqie/fuOHLkiOZxlUqFgQMHoly5cjAyMkLlypWxZMkSzeMzZszAxo0b8eeff2pGgU6ePAkAePz4MTw8PGBpaYmiRYuic+fOePjw4UfzEBUWLG6IChm5XI6lS5fixo0b2LhxI44fP47x48dn2L93794oXbo0Ll26hMuXL2PixIkwMDAAAISEhKBt27bo2rUr/vvvP/j5+eHMmTMYPny4TpmMjIygVquRkpKCJUuWYMGCBfjll1/w33//wc3NDZ06dcLdu3cBAEuXLsXevXvh7++P27dvY+vWrbC3t093vRcvXgQAHD16FGFhYdizZ0+aPt27d8fLly9x4sQJTdurV68QEBCA3r17AwBOnz6Nvn37YtSoUbh58yZWr14NX19fzJ49O9Ov8eHDhzh8+DAUCoWmTa1Wo3Tp0ti5cydu3ryJ6dOnY/LkyfD39wcAjB07Fh4eHmjbti3CwsIQFhaGhg0bIjk5GW5ubjAzM8Pp06dx9uxZmJqaom3btkhKSsp0JqICK8cvzUlEuc7Ly0vo6ekJExMTza1bt27p9t25c6coVqyY5v6GDRuEhYWF5r6ZmZnw9fVNd9mBAweKwYMHa7WdPn1ayOVyER8fn+4yH67/zp07olKlSqJu3bpCCCFKliwpZs+erbVMvXr1xNChQ4UQQowYMUK0bNlSqNXqdNcPQPz+++9CCCEePHggAIgrV65o9fnwiuadO3cWAwYM0NxfvXq1KFmypFCpVEIIIVq1aiXmzJmjtY7NmzcLW1vbdDMIIYS3t7eQy+XCxMREGBoaaq6evHDhwgyXEUKIYcOGia5du2aYNfW5K1eurLUNEhMThZGRkTh8+PBH109UGHDODVEB1aJFC6xcuVJz38TEBMC7UQwfHx8EBwcjOjoaKSkpSEhIwNu3b2FsbJxmPWPGjME333yDzZs3a3atlC9fHsC7XVb//fcftm7dqukvhIBarcaDBw9QtWrVdLNFRUXB1NQUarUaCQkJaNy4MdatW4fo6Gg8e/YMjRo10urfqFEjXL16FcC7XUqtW7dG5cqV0bZtW3Ts2BFt2rT5rG3Vu3dvDBo0CCtWrIBSqcTWrVvRo0cPyOVyzes8e/as1kiNSqX66HYDgMqVK2Pv3r1ISEjAli1bEBQUhBEjRmj1Wb58OdavX4/Q0FDEx8cjKSkJjo6OH8179epV3Lt3D2ZmZlrtCQkJCAkJycIWICpYWNwQFVAmJiaoUKGCVtvDhw/RsWNHDBkyBLNnz0bRokVx5swZDBw4EElJSel+SM+YMQO9evXCgQMHcOjQIXh7e2PHjh348ssvERsbi2+//RYjR45Ms1yZMmUyzGZmZobAwEDI5XLY2trCyMgIABAdHf3J1+Xk5IQHDx7g0KFDOHr0KDw8PODq6opdu3Z9ctmMuLu7QwiBAwcOoF69ejh9+jQWLVqkeTw2NhYzZ87EV199lWZZQ0PDDNerUCg0P4O5c+eiQ4cOmDlzJn766ScAwI4dOzB27FgsWLAADRo0gJmZGebPn48LFy58NG9sbCycnZ21ispUeWXSOJGUWNwQFSKXL1+GWq3GggULNKMSqfM7PqZSpUqoVKkSRo8ejZ49e2LDhg348ssv4eTkhJs3b6Ypoj5FLpenu4y5uTlKliyJs2fPolmzZpr2s2fPon79+lr9PD094enpiW7duqFt27Z49eoVihYtqrW+1PktKpXqo3kMDQ3x1VdfYevWrbh37x4qV64MJycnzeNOTk64ffu2zq/zQ1OnTkXLli0xZMgQzets2LAhhg4dqunz4ciLQqFIk9/JyQl+fn6wtraGubn5Z2UiKog4oZioEKlQoQKSk5Px66+/4v79+9i8eTNWrVqVYf/4+HgMHz4cJ0+exKNHj3D27FlcunRJs7tpwoQJOHfuHIYPH46goCDcvXsXf/75p84Tit83btw4zJs3D35+frh9+zYmTpyIoKAgjBo1CgCwcOFCbN++HcHBwbhz5w527tyJEiVKpHviQWtraxgZGSEgIAARERGIiorK8Hl79+6NAwcOYP369ZqJxKmmT5+OTZs2YebMmbhx4wZu3bqFHTt2YOrUqTq9tgYNGqBWrVqYM2cOAKBixYr4999/cfjwYdy5cwfTpk3DpUuXtJaxt7fHf//9h9u3byMyMhLJycno3bs3rKys0LlzZ5w+fRoPHjzAyZMnMXLkSDx58kSnTEQFktSTfogo+6U3CTXVwoULha2trTAyMhJubm5i06ZNAoB4/fq1EEJ7wm9iYqLo0aOHsLOzEwqFQpQsWVIMHz5ca7LwxYsXRevWrYWpqakwMTERtWrVSjMh+H0fTij+kEqlEjNmzBClSpUSBgYGonbt2uLQoUOax9esWSMcHR2FiYmJMDc3F61atRKBgYGax/HehGIhhFi7dq2ws7MTcrlcNGvWLMPto1KphK2trQAgQkJC0uQKCAgQDRs2FEZGRsLc3FzUr19frFmzJsPX4e3tLWrXrp2mffv27UKpVIrQ0FCRkJAg+vXrJywsLISlpaUYMmSImDhxotZyz58/12xfAOLEiRNCCCHCwsJE3759hZWVlVAqlcLBwUEMGjRIREVFZZiJqLCQCSGEtOUVERERUfbhbikiIiIqUFjcEBERUYHC4oaIiIgKFBY3REREVKCwuCEiIqIChcUNERERFSgsboiIiKhAYXFDREREBQqLGyIiIipQWNwQERFRgcLihoiIiAoUFjdERERUoPwf6DQMRAXFijYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot perfect ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_test)\n",
    "plot_roc_curve(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(1.0)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perfect ROC AUC score\n",
    "roc_auc_score(y_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality, a perfect ROC curve is unlikely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix\n",
    "\n",
    "Another fantastic way to evaluate a classification model is by using a [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix). \n",
    "\n",
    "A confusion matrix is a quick way to compare the labels a model predicts and the actual labels it was supposed to predict. \n",
    "\n",
    "In essence, giving you an idea of where the model is getting confused."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[24,  5],\n",
       "       [ 4, 28]])"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "y_preds = clf.predict(X_test)\n",
    "\n",
    "confusion_matrix(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this is probably easier visualized.\n",
    "\n",
    "One way to do it is with `pd.crosstab()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted Label</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual Label</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>24</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted Label   0   1\n",
       "Actual Label           \n",
       "0                24   5\n",
       "1                 4  28"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(y_test, \n",
    "            y_preds, \n",
    "            rownames=[\"Actual Label\"], \n",
    "            colnames=[\"Predicted Label\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a confusion matrix using Scikit-Learn\n",
    "\n",
    "Scikit-Learn has multiple different implementations of plotting confusion matrices:\n",
    "\n",
    "1. [`sklearn.metrics.ConfusionMatrixDisplay.from_estimator(estimator, X, y)`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay.from_estimator) - this takes a fitted estimator (like our `clf` model), features (`X`) and labels (`y`), it then uses the trained estimator to make predictions on `X` and compares the predictions to `y` by displaying a confusion matrix.\n",
    "2. [`sklearn.metrics.ConfusionMatrixDisplay.from_predictions(y_true, y_pred)`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay.from_predictions) - this takes truth labels and predicted labels and compares them by displaying a confusion matrix.\n",
    "\n",
    "> **Note:** Both of these methods/classes require Scikit-Learn 1.0+. To check your version of Scikit-Learn run:\n",
    "```python\n",
    "import sklearn\n",
    "sklearn.__version__\n",
    "```\n",
    "> If you don't have 1.0+, you can upgrade at: https://scikit-learn.org/stable/install.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAG0CAYAAAA1hY5rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA2iklEQVR4nO3dfVhUdf7/8dcMyIDKgFiCFCit92VqWkR3arGRtaar+23tSy2Z6bdNNKW8+22Y2g1lN5JG0q3m/nS72dJvWWs/FzM0yRKz3+YSpVGSCm4/QgTlxpnz+4N1+k5qMczAOHOej+s61+V8zvmc8x4vLt68P5/POcdiGIYhAAAQtKz+DgAAALQtkj0AAEGOZA8AQJAj2QMAEORI9gAABDmSPQAAQY5kDwBAkCPZAwAQ5Ej2AAAEOZI9AABBjmQPAEAbKCws1OjRoxUfHy+LxaJ169addExJSYluvPFGRUVFqVOnTrr44ou1b98+1/76+npNnTpVXbt2VefOnTV+/HhVVlZ6HEuoN1/E35xOpw4cOKDIyEhZLBZ/hwMA8JBhGDpy5Iji4+NltbZd/VlfX6/GxkavzxMWFqbw8PAWHVtXV6dBgwbp9ttv17hx407av3fvXl1xxRWaNGmSFi5cKLvdrt27d7udf+bMmXrnnXf0+uuvKyoqSpmZmRo3bpw+/PBDzwI3Alh5ebkhiY2NjY0twLfy8vI2yxXHjh0z4rqF+CTOuLg449ixYx7HIMlYu3atW9vvf/9745Zbbjltn+rqaqNDhw7G66+/7morKSkxJBlFRUUeXT+gK/vIyEhJ0tyC4QrvHNBfBTitzZdG+jsEoM0cN5q0Ve+4fp+3hcbGRlUccujb4p6yR7Z+9KDmiFM9hn6j77//Xna73dVus9lks9k8OpfT6dQ777yj2bNnKy0tTZ9++qmSkpI0b948jR07VpJUXFyspqYmpaamuvr169dPiYmJKioq0qWXXtri6wV0hjwxdB/eOZRkj6AVaung7xCAtmWoXaZiO0da1Dmy9ddxqrlvQkKCW/v999+vBQsWeHSuQ4cOqba2Vo888ogefPBBPfroo9qwYYPGjRun999/X8OHD1dFRYXCwsIUHR3t1jc2NlYVFRUeXY8MCQAwBYfhlMPwrr8klZeXn1TZe8rpbD7XmDFjNHPmTEnS4MGDtW3bNuXn52v48OGtD/QUSPYAAFNwypBTrc/2J/ra7Xa3ZN8aZ511lkJDQzVgwAC39v79+2vr1q2SpLi4ODU2Nqq6utqtuq+srFRcXJxH1+PWOwAA2llYWJguvvhilZaWurV/+eWX6tGjhyRp6NCh6tChgwoKClz7S0tLtW/fPqWkpHh0PSp7AIApOOWU08v+nqitrdWePXtcn8vKyrRr1y7FxMQoMTFRs2bN0u9//3tdddVVGjlypDZs2KC3335bmzdvliRFRUVp0qRJysrKUkxMjOx2u6ZNm6aUlBSPFudJJHsAgEk4DEMOo/XD+J723bFjh0aOHOn6nJWVJUnKyMjQypUr9dvf/lb5+fnKycnR9OnT1bdvX73xxhu64oorXH2WLFkiq9Wq8ePHq6GhQWlpaXrmmWc8jt3y7/v/AlJNTY2ioqK0YPs1rMZH0Pr7QO/mBoEz2XGjSZuNdTp8+LDX8+CncyJXlH9xjte33iX029+msbYVMiQAwBR8tUAvEJHsAQCm4JQhh0mTPavxAQAIclT2AABTYBgfAIAg196r8c8kDOMDABDkqOwBAKbg/PfmTf9ARbIHAJiCw8vV+N709TeSPQDAFByGvHzrne9iaW/M2QMAEOSo7AEApsCcPQAAQc4pixyyeNU/UDGMDwBAkKOyBwCYgtNo3rzpH6hI9gAAU3B4OYzvTV9/YxgfAIAgR2UPADAFM1f2JHsAgCk4DYuchher8b3o628M4wMAEOSo7AEApsAwPgAAQc4hqxxeDGg7fBhLeyPZAwBMwfByzt5gzh4AAJypqOwBAKbAnD0AAEHOYVjlMLyYsw/gx+UyjA8AQJCjsgcAmIJTFjm9qHGdCtzSnmQPADAFM8/ZM4wPAECQo7IHAJiC9wv0GMYHAOCM1jxn78WLcBjGBwAAZyoqewCAKTi9fDY+q/EBADjDMWcPAECQc8pq2vvsmbMHACDIkewBAKbgMCxeb54oLCzU6NGjFR8fL4vFonXr1p322DvvvFMWi0W5ublu7VVVVUpPT5fdbld0dLQmTZqk2tpaj787yR4AYAqOfy/Q82bzRF1dnQYNGqS8vLyfPW7t2rX66KOPFB8ff9K+9PR07d69Wxs3btT69etVWFioKVOmeBSHxJw9AABtYtSoURo1atTPHrN//35NmzZN7733nm644Qa3fSUlJdqwYYM++eQTDRs2TJK0bNkyXX/99Xr88cdP+cfB6VDZAwBMwWlYvd58Go/TqVtvvVWzZs3S+eeff9L+oqIiRUdHuxK9JKWmpspqtWr79u0eXYvKHgBgCq0Zinfv37wav6amxq3dZrPJZrN5fL5HH31UoaGhmj59+in3V1RUqFu3bm5toaGhiomJUUVFhUfXorIHAMADCQkJioqKcm05OTken6O4uFhPPfWUVq5cKYul7R/DS2UPADAFp+Txivqf9pek8vJy2e12V3trqvotW7bo0KFDSkxMdLU5HA7dc889ys3N1TfffKO4uDgdOnTIrd/x48dVVVWluLg4j65HsgcAmIL3D9Vp7mu3292SfWvceuutSk1NdWtLS0vTrbfeqokTJ0qSUlJSVF1dreLiYg0dOlSStGnTJjmdTiUnJ3t0PZI9AABtoLa2Vnv27HF9Lisr065duxQTE6PExER17drV7fgOHTooLi5Offv2lST1799f1113nSZPnqz8/Hw1NTUpMzNTEyZM8GglvkSyBwCYhPfPxves744dOzRy5EjX56ysLElSRkaGVq5c2aJzrF69WpmZmbrmmmtktVo1fvx4LV261KM4JJI9AMAk2vt99iNGjJDhwctzvvnmm5PaYmJitGbNGo+ueyokewCAKbR3ZX8mCdzIAQBAi1DZAwBMwfuH6gRufUyyBwCYgtOwyOnNffZe9PW3wP0zBQAAtAiVPQDAFJxeDuN780AefyPZAwBMwds31/n6rXftKXAjBwAALUJlDwAwBYcscnjxUB1v+vobyR4AYAoM4wMAgKBFZQ8AMAWHvBuKd/gulHZHsgcAmIKZh/FJ9gAAU+BFOAAAIGhR2QMATMHw8n32BrfeAQBwZmMYHwAABC0qewCAKZj5FbckewCAKTi8fOudN339LXAjBwAALUJlDwAwBYbxAQAIck5Z5fRiQNubvv4WuJEDAIAWobIHAJiCw7DI4cVQvDd9/Y1kDwAwBebsAQAIcoaXb70zeIIeAAA4U1HZAwBMwSGLHF68zMabvv5GsgcAmILT8G7e3Wn4MJh2xjA+AABBjsoe+mFHiL5dEaaaf1rV+C+rLnzqmLpdc9y1f29emCo3hKq+wiprB8k+wKFfTW9Q1IVO1zG7MiN05AurmqosCrUbirnUod5ZDbJ1C+A/hWEat2Qd1K33VLq1le+x6Y7h/f0UEdqC08sFet709bczIvK8vDz17NlT4eHhSk5O1scff+zvkEzFcUzq3Nehfn9qOOX+Tj2d6vu/GnTpm3UatuqowuOd2jmloxqrfhwO63LJcV34xDGlrK/ThUuO6Vi5Vf93ZkR7fQXAa998Ea4Jg893bVlje/s7JPiYUxavt0Dl98r+1VdfVVZWlvLz85WcnKzc3FylpaWptLRU3bp183d4pnDWlQ6ddaXjtPvjbjj+Pz4Z6jO7QQfeDFPtl1bFXNrcr8cfmlxHRMQb6nlHgz6bHiFnk2Tt0FaRA77jcEg//IsfVgQnv1f2Tz75pCZPnqyJEydqwIABys/PV8eOHfXSSy/5OzScgrNJ2v96B4VGGurc13nKY5oOSxXrOyhqsINEj4BxTlKj1hR/rpXb/qk5y77V2fGN/g4JPnbiCXrebIHKr5V9Y2OjiouLNW/ePFeb1WpVamqqioqK/BgZfupfm0P0+awIOeol29mGhjx3VGFd3Ofjv3oyTOV/CZPzmEVRgxwalHfUT9ECnvni0056fGaEvttrU0y3Jt2SVaEn1n6l/7q6n47Vhfg7PPgIc/Z+8v3338vhcCg2NtatPTY2VhUVFScd39DQoJqaGrcN7SPmEoeS36jTxf/7qLpeflz/uDdCjf/P/a/cHhOblPx6nYY8d1SyGto9L0IG6/MQAHa8b9eW9dEqK4lQ8Qd23Xfreepsd+iq0dX+Dg0BrLCwUKNHj1Z8fLwsFovWrVvn2tfU1KQ5c+Zo4MCB6tSpk+Lj4/WHP/xBBw4ccDtHVVWV0tPTZbfbFR0drUmTJqm2ttbjWALqz5ScnBxFRUW5toSEBH+HZBohHaWOiYaiBjk14IEGWUKk/W+6j9GHdTHUqaehrpc5NPCxev2/LaE6/FlA/YgBkqS6mlB997VN8T1PvWgVgckpi+v5+K3aPFygV1dXp0GDBikvL++kfUePHtXOnTuVnZ2tnTt36s0331RpaaluvPFGt+PS09O1e/dubdy4UevXr1dhYaGmTJni8Xf36zD+WWedpZCQEFVWut/yUllZqbi4uJOOnzdvnrKyslyfa2pqSPj+4pScPzel+e+K3tkYuHNcMK/wjg7F92hUwRssOgkmhpcr6g0P+44aNUqjRo065b6oqCht3LjRre3pp5/WJZdcon379ikxMVElJSXasGGDPvnkEw0bNkyStGzZMl1//fV6/PHHFR8f3+JY/Fp2hYWFaejQoSooKHC1OZ1OFRQUKCUl5aTjbTab7Ha72wbvHT8qHfnCqiNfNP84HNtv0ZEvrKo/aJHjqLQnN0yHP7Pq2AGLanZbtfu+cDUcsig2rXmV/uH/a1X5mg468kXzMVXbQ/SPWRGKSHAqevDpV/kDZ4rJ2fs18NJaxZ7boAHD6nT/i2VyOKXN67r4OzT4kFdVvZdvzGuJw4cPy2KxKDo6WpJUVFSk6OhoV6KXpNTUVFmtVm3fvt2jc/v91rusrCxlZGRo2LBhuuSSS5Sbm6u6ujpNnDjR36GZRs3nIdp5e0fX568Wh0uSuo9pUr/59aors+rgWxFq/MGiDtGG7Bc4NPTlo+rcq3k1fki4dOjvofo6zybHMSnsbENdLz+upP9qlDXML18J8MhZ3Zs0L+8bRXZx6HBVqHZ/3EkzRvfR4Sq//4rEGein68VsNptsNptX56yvr9ecOXN08803uwrZioqKk25BDw0NVUxMzCnXtf0cv/8k//73v9e//vUvzZ8/XxUVFRo8eLA2bNhw0qI9tJ2YSxxK/fzIafcPeqr+Z/t37uPU0JeO+TosoN3k3NXT3yGgHfhqNf5Pp4/vv/9+LViwoNXnbWpq0k033STDMLR8+fJWn+fn+D3ZS1JmZqYyMzP9HQYAIIh5OxR/om95ebnbNLI3Vf2JRP/tt99q06ZNbueNi4vToUOH3I4/fvy4qqqqTrmu7eewVBoAAA/8dO1Ya5P9iUT/1Vdf6e9//7u6du3qtj8lJUXV1dUqLi52tW3atElOp1PJyckeXeuMqOwBAGhr3j7f3tO+tbW12rNnj+tzWVmZdu3apZiYGHXv3l2/+93vtHPnTq1fv14Oh8M1Dx8TE6OwsDD1799f1113nSZPnqz8/Hw1NTUpMzNTEyZM8GglvkSyBwCYhK+G8Vtqx44dGjlypOvziVvHMzIytGDBAr311luSpMGDB7v1e//99zVixAhJ0urVq5WZmalrrrlGVqtV48eP19KlSz2OnWQPAEAbGDFihIyfeYzoz+07ISYmRmvWrPE6FpI9AMAU2ruyP5OQ7AEApmDmZM9qfAAAghyVPQDAFMxc2ZPsAQCmYMjz2+d+2j9QkewBAKZg5sqeOXsAAIIclT0AwBTMXNmT7AEApmDmZM8wPgAAQY7KHgBgCmau7En2AABTMAyLDC8Stjd9/Y1hfAAAghyVPQDAFNr7ffZnEpI9AMAUzDxnzzA+AABBjsoeAGAKZl6gR7IHAJiCmYfxSfYAAFMwc2XPnD0AAEGOyh4AYAqGl8P4gVzZk+wBAKZgSDIM7/oHKobxAQAIclT2AABTcMoiC0/QAwAgeLEaHwAABC0qewCAKTgNiyw8VAcAgOBlGF6uxg/g5fgM4wMAEOSo7AEApmDmBXokewCAKZDsAQAIcmZeoMecPQAAQY7KHgBgCmZejU+yBwCYQnOy92bO3ofBtDOG8QEACHIkewCAKZxYje/N5onCwkKNHj1a8fHxslgsWrdu3U/iMTR//nx1795dERERSk1N1VdffeV2TFVVldLT02W32xUdHa1JkyaptrbW4+9OsgcAmILhg80TdXV1GjRokPLy8k65f/HixVq6dKny8/O1fft2derUSWlpaaqvr3cdk56ert27d2vjxo1av369CgsLNWXKFA8jYc4eAIA2MWrUKI0aNeqU+wzDUG5uru677z6NGTNGkrRq1SrFxsZq3bp1mjBhgkpKSrRhwwZ98sknGjZsmCRp2bJluv766/X4448rPj6+xbFQ2QMATKG9h/F/TllZmSoqKpSamupqi4qKUnJysoqKiiRJRUVFio6OdiV6SUpNTZXVatX27ds9uh6VPQDAHFozFv/T/pJqamrcmm02m2w2m0enqqiokCTFxsa6tcfGxrr2VVRUqFu3bm77Q0NDFRMT4zqmpajsAQDm4G1V/+/KPiEhQVFRUa4tJyfHz1/sl1HZAwDggfLyctntdtdnT6t6SYqLi5MkVVZWqnv37q72yspKDR482HXMoUOH3PodP35cVVVVrv4tRWUPADCFE0/Q82aTJLvd7ra1JtknJSUpLi5OBQUFrraamhpt375dKSkpkqSUlBRVV1eruLjYdcymTZvkdDqVnJzs0fWo7AEAptDeb72rra3Vnj17XJ/Lysq0a9cuxcTEKDExUTNmzNCDDz6o3r17KykpSdnZ2YqPj9fYsWMlSf3799d1112nyZMnKz8/X01NTcrMzNSECRM8WokvkewBAGgTO3bs0MiRI12fs7KyJEkZGRlauXKlZs+erbq6Ok2ZMkXV1dW64oortGHDBoWHh7v6rF69WpmZmbrmmmtktVo1fvx4LV261ONYSPYAAHP4H4vsWt3fAyNGjJDxMw/Ut1gsWrRokRYtWnTaY2JiYrRmzRqPrnsqJHsAgCmY+a13LNADACDIUdkDAMzBRw/VCUQtSvZvvfVWi0944403tjoYAADaSnuvxj+TtCjZn7gN4JdYLBY5HA5v4gEAAD7WomTvdDrbOg4AANpeAA/Fe8OrOfv6+nq3+wEBADhTmXkY3+PV+A6HQw888IDOOeccde7cWV9//bUkKTs7Wy+++KLPAwQAwCcMH2wByuNk/9BDD2nlypVavHixwsLCXO0XXHCBXnjhBZ8GBwAAvOdxsl+1apWee+45paenKyQkxNU+aNAgffHFFz4NDgAA37H4YAtMHs/Z79+/X7169Tqp3el0qqmpySdBAQDgcya+z97jyn7AgAHasmXLSe1//etfNWTIEJ8EBQAAfMfjyn7+/PnKyMjQ/v375XQ69eabb6q0tFSrVq3S+vXr2yJGAAC8R2XfcmPGjNHbb7+tv//97+rUqZPmz5+vkpISvf322/r1r3/dFjECAOC9E2+982YLUK26z/7KK6/Uxo0bfR0LAABoA61+qM6OHTtUUlIiqXkef+jQoT4LCgAAXzPzK249Tvbfffedbr75Zn344YeKjo6WJFVXV+uyyy7TK6+8onPPPdfXMQIA4D3m7FvujjvuUFNTk0pKSlRVVaWqqiqVlJTI6XTqjjvuaIsYAQCAFzyu7D/44ANt27ZNffv2dbX17dtXy5Yt05VXXunT4AAA8BlvF9mZaYFeQkLCKR+e43A4FB8f75OgAADwNYvRvHnTP1B5PIz/2GOPadq0adqxY4erbceOHbr77rv1+OOP+zQ4AAB8xsQvwmlRZd+lSxdZLD8OX9TV1Sk5OVmhoc3djx8/rtDQUN1+++0aO3ZsmwQKAABap0XJPjc3t43DAACgjTFn//MyMjLaOg4AANqWiW+9a/VDdSSpvr5ejY2Nbm12u92rgAAAgG95vECvrq5OmZmZ6tatmzp16qQuXbq4bQAAnJFMvEDP42Q/e/Zsbdq0ScuXL5fNZtMLL7yghQsXKj4+XqtWrWqLGAEA8J6Jk73Hw/hvv/22Vq1apREjRmjixIm68sor1atXL/Xo0UOrV69Wenp6W8QJAABayePKvqqqSuedd56k5vn5qqoqSdIVV1yhwsJC30YHAICvmPgVtx4n+/POO09lZWWSpH79+um1116T1Fzxn3gxDgAAZ5oTT9DzZgtUHif7iRMn6rPPPpMkzZ07V3l5eQoPD9fMmTM1a9YsnwcIAAC84/Gc/cyZM13/Tk1N1RdffKHi4mL16tVLF154oU+DAwDAZ7jPvvV69OihHj16+CIWAADQBlqU7JcuXdriE06fPr3VwQAA0FYs8vKtdz6LpP21KNkvWbKkRSezWCwkewAAzjAtSvYnVt+fqTYnd1aopYO/wwDaxHsHPvV3CECbqTniVJc+7XQxE78Ix+PV+AAABKR2foKew+FQdna2kpKSFBERoV/96ld64IEHZBg/nsgwDM2fP1/du3dXRESEUlNT9dVXX3n5RU9GsgcAoA08+uijWr58uZ5++mmVlJTo0Ucf1eLFi7Vs2TLXMYsXL9bSpUuVn5+v7du3q1OnTkpLS1N9fb1PY/F6NT4AAAGhnW+927Ztm8aMGaMbbrhBktSzZ0/95S9/0ccff9x8OsNQbm6u7rvvPo0ZM0aStGrVKsXGxmrdunWaMGGCF8G6o7IHAJhCez9B77LLLlNBQYG+/PJLSdJnn32mrVu3atSoUZKa18NVVFQoNTXV1ScqKkrJyckqKiry2feWqOwBAPBITU2N22ebzSabzXbScXPnzlVNTY369eunkJAQORwOPfTQQ64XxlVUVEiSYmNj3frFxsa69vlKqyr7LVu26JZbblFKSor2798vSfrzn/+srVu3+jQ4AAB8xkcL9BISEhQVFeXacnJyTnm51157TatXr9aaNWu0c+dOvfzyy3r88cf18ssvt+GXPDWPK/s33nhDt956q9LT0/Xpp5+qoaFBknT48GE9/PDDevfdd30eJAAAXvPRnH15ebnsdrur+VRVvSTNmjVLc+fOdc29Dxw4UN9++61ycnKUkZGhuLg4SVJlZaW6d+/u6ldZWanBgwd7EejJPK7sH3zwQeXn5+v5559Xhw4/3tt++eWXa+fOnT4NDgCAM43dbnfbTpfsjx49KqvVPc2GhITI6XRKkpKSkhQXF6eCggLX/pqaGm3fvl0pKSk+jdnjyr60tFRXXXXVSe1RUVGqrq72RUwAAPict6+p9bTv6NGj9dBDDykxMVHnn3++Pv30Uz355JO6/fbbm89nsWjGjBl68MEH1bt3byUlJSk7O1vx8fEaO3Zs6wM9BY+TfVxcnPbs2aOePXu6tW/dulXnnXeer+ICAMC32vkJesuWLVN2drbuuusuHTp0SPHx8fqv//ovzZ8/33XM7NmzVVdXpylTpqi6ulpXXHGFNmzYoPDw8NbHeQoeJ/vJkyfr7rvv1ksvvSSLxaIDBw6oqKhI9957r7Kzs30aHAAAPtPO99lHRkYqNzdXubm5pz3GYrFo0aJFWrRokReB/TKPk/3cuXPldDp1zTXX6OjRo7rqqqtks9l07733atq0aW0RIwAA8ILHyd5isehPf/qTZs2apT179qi2tlYDBgxQ586d2yI+AAB8or3n7M8krX6oTlhYmAYMGODLWAAAaDvtPIx/JvE42Y8cOVIWy+kXKWzatMmrgAAAgG95nOx/eqN/U1OTdu3apc8//1wZGRm+igsAAN/ychjfVJX9kiVLTtm+YMEC1dbWeh0QAABtwsTD+D57690tt9yil156yVenAwAAPuKzt94VFRX5/CEAAAD4jIkre4+T/bhx49w+G4ahgwcPaseOHTxUBwBwxuLWOw9ERUW5fbZarerbt68WLVqka6+91meBAQAA3/Ao2TscDk2cOFEDBw5Uly5d2iomAADgQx4t0AsJCdG1117L2+0AAIHH8MEWoDxejX/BBRfo66+/botYAABoMyfm7L3ZApXHyf7BBx/Uvffeq/Xr1+vgwYOqqalx2wAAwJmlxXP2ixYt0j333KPrr79eknTjjTe6PTbXMAxZLBY5HA7fRwkAgC8EcHXujRYn+4ULF+rOO+/U+++/35bxAADQNrjP/pcZRvO3HD58eJsFAwAAfM+jW+9+7m13AACcyXioTgv16dPnFxN+VVWVVwEBANAmGMZvmYULF570BD0AAHBm8yjZT5gwQd26dWurWAAAaDMM47cA8/UAgIBm4mH8Fj9U58RqfAAAEFhaXNk7nc62jAMAgLZl4sre41fcAgAQiJizBwAg2Jm4svf4RTgAACCwUNkDAMzBxJU9yR4AYApmnrNnGB8AgCBHZQ8AMAeG8QEACG4M4wMAgKBFZQ8AMAeG8QEACHImTvYM4wMAEORI9gAAU7D4YPPU/v37dcstt6hr166KiIjQwIEDtWPHDtd+wzA0f/58de/eXREREUpNTdVXX33V+i95GiR7AIA5GD7YPPDDDz/o8ssvV4cOHfS3v/1N//znP/XEE0+oS5curmMWL16spUuXKj8/X9u3b1enTp2Ulpam+vp6L7+sO+bsAQCm0N633j366KNKSEjQihUrXG1JSUmufxuGodzcXN13330aM2aMJGnVqlWKjY3VunXrNGHChNYH+xNU9gAAtIG33npLw4YN03/8x3+oW7duGjJkiJ5//nnX/rKyMlVUVCg1NdXVFhUVpeTkZBUVFfk0FpI9AMAcfDSMX1NT47Y1NDSc8nJff/21li9frt69e+u9997TH//4R02fPl0vv/yyJKmiokKSFBsb69YvNjbWtc9XSPYAAPPwwXx9QkKCoqKiXFtOTs4pL+V0OnXRRRfp4Ycf1pAhQzRlyhRNnjxZ+fn5bfb1Toc5ewAAPFBeXi673e76bLPZTnlc9+7dNWDAALe2/v3764033pAkxcXFSZIqKyvVvXt31zGVlZUaPHiwT2OmsgcAmMKJBXrebJJkt9vdttMl+8svv1ylpaVubV9++aV69OghqXmxXlxcnAoKClz7a2pqtH37dqWkpPj0u1PZAwDMoZ2foDdz5kxddtllevjhh3XTTTfp448/1nPPPafnnntOkmSxWDRjxgw9+OCD6t27t5KSkpSdna34+HiNHTvWi0BPRrIHAKANXHzxxVq7dq3mzZunRYsWKSkpSbm5uUpPT3cdM3v2bNXV1WnKlCmqrq7WFVdcoQ0bNig8PNynsZDsAQCm4I9X3P7mN7/Rb37zm9Of02LRokWLtGjRotYH1gIkewCAOfAiHAAAEKyo7AEApuCPYfwzBckeAGAOJh7GJ9kDAMzBxMmeOXsAAIIclT0AwBSYswcAINgxjA8AAIIVlT0AwBQshiGL0fry3Ju+/kayBwCYA8P4AAAgWFHZAwBMgdX4AAAEO4bxAQBAsKKyBwCYAsP4AAAEOxMP45PsAQCmYObKnjl7AACCHJU9AMAcGMYHACD4BfJQvDcYxgcAIMhR2QMAzMEwmjdv+gcokj0AwBRYjQ8AAIIWlT0AwBxYjQ8AQHCzOJs3b/oHKobx4bGbMiv13oHPdOfC/f4OBWiRf3zUSfP/kKSbh5yvtPjB2va3qJOO2feVTfdnJOm3fQfqxl8N1LRRfXTouw6u/e/+766aNb6XfttnoNLiB6v2cEh7fgXAKyR7eKTPoKO64ZYqfb073N+hAC1Wf9Sq884/psyHvzvl/gPfhClrbG8l9KrXY3/do/yCUv3njAqFhf84blt/zKphI2o0YVple4UNXzN8sAUovw7jFxYW6rHHHlNxcbEOHjyotWvXauzYsf4MCT8jvKNDc57+VrmzztXNd/MLD4Hj4quP6OKrj5x2/8pHuuuSq2t0R/ZBV1t8z0a3Y8ZN/pck6bNtndsmSLQ5VuP7SV1dnQYNGqS8vDx/hoEWynx4vz4usOvTLZH+DgXwGadT+rjArnPOa9D/uvk83TTwfE2/ofcph/oR4E7cZ+/NFqD8WtmPGjVKo0aN8mcIaKHhY35Qr4HHNO363v4OBfCp6u9DdawuRK8+3U23zanQpD8d1I73I7Xojp5a/Nc9ujClzt8hAl4LqNX4DQ0NamhocH2uqanxYzTmcXZ8o/646IDmTThPTQ0s80BwMf69wjolrUbjpjQP1f/qgmP6545OemfVWST7IGLmYfyASvY5OTlauHChv8MwnV4XHlOXs48r770vXW0hodLAS+t048Tv9ZueF8rptPgxQqD17DEOhYQa6tGn3q09oXe9dn/cyU9RoU1wn31gmDdvnrKyslyfa2pqlJCQ4MeIzGHXls6aMrKPW9s9S8pVvidcr+WdTaJHQOsQZqjPoKP6bq/NrX3/1zZ1O7fJT1EBvhVQyd5ms8lms/3ygfCpY3Uh+rY0wq2t/qhVR344uR04Ex2rs+pA2Y+/OyrKw7T38whFRh9Xt3Ob9B93HdLDd/bQBZfWatBltdrxvl0fbYzSY3/d4+pTdShUPxzqoANlYZKksi/C1bGTU2ef0yh7F0e7fyd4zszD+EzAAgh6X37WUXdd21d3XdtXkvTsgnN017V9terx7pKky0cd1vRHvtPrz8Tqzmv6acOarsp+vkwXJP84X//OqrN017V9lTsrUZJ07297665r++qj/8Oq/YDhx9X4jzzyiCwWi2bMmOFqq6+v19SpU9W1a1d17txZ48ePV2Vl29zW7NfKvra2Vnv2/PiXc1lZmXbt2qWYmBglJib6MTL8ktm/6+XvEIAWG3RZrd47sOtnj0m7uUppN1eddv+t91bo1nsrfBwZzOCTTz7Rs88+qwsvvNCtfebMmXrnnXf0+uuvKyoqSpmZmRo3bpw+/PBDn8fg18p+x44dGjJkiIYMGSJJysrK0pAhQzR//nx/hgUACEInhvG92TxVW1ur9PR0Pf/88+rSpYur/fDhw3rxxRf15JNP6uqrr9bQoUO1YsUKbdu2TR999JEPv3Uzvyb7ESNGyDCMk7aVK1f6MywAQDDyw+Nyp06dqhtuuEGpqalu7cXFxWpqanJr79evnxITE1VUVOT5hX5BQC3QAwDA3376jJfTLR5/5ZVXtHPnTn3yyScn7auoqFBYWJiio6Pd2mNjY1VR4fvpIhboAQBMwVfD+AkJCYqKinJtOTk5J12rvLxcd999t1avXq3wcP+/OIzKHgBgDk6jefOmv5oTud1udzWfqqovLi7WoUOHdNFFF7naHA6HCgsL9fTTT+u9995TY2Ojqqur3ar7yspKxcXFtT7G0yDZAwDMwUdP0LPb7W7J/lSuueYa/eMf/3Brmzhxovr166c5c+YoISFBHTp0UEFBgcaPHy9JKi0t1b59+5SSkuJFkKdGsgcAwMciIyN1wQUXuLV16tRJXbt2dbVPmjRJWVlZiomJkd1u17Rp05SSkqJLL73U5/GQ7AEApmCRl0/Q81kkzZYsWSKr1arx48eroaFBaWlpeuaZZ3x8lWYkewCAOXj7Tnov32e/efNmt8/h4eHKy8tTXl6eV+dtCVbjAwAQ5KjsAQCmYOYX4ZDsAQDmYOL32TOMDwBAkKOyBwCYgsUwZPFikZ03ff2NZA8AMAfnvzdv+gcohvEBAAhyVPYAAFNgGB8AgGBn4tX4JHsAgDn4+Ql6/sScPQAAQY7KHgBgCjxBDwCAYMcwPgAACFZU9gAAU7A4mzdv+gcqkj0AwBwYxgcAAMGKyh4AYA48VAcAgOBm5sflMowPAECQo7IHAJiDiRfokewBAOZgyLt30gdurifZAwDMgTl7AAAQtKjsAQDmYMjLOXufRdLuSPYAAHMw8QI9hvEBAAhyVPYAAHNwSrJ42T9AkewBAKbAanwAABC0qOwBAOZg4gV6JHsAgDmYONkzjA8AQJCjsgcAmIOJK3uSPQDAHLj1DgCA4MatdwAAIGiR7AEA5nBizt6bzQM5OTm6+OKLFRkZqW7dumns2LEqLS11O6a+vl5Tp05V165d1blzZ40fP16VlZW+/NaSSPYAALNwGt5vHvjggw80depUffTRR9q4caOampp07bXXqq6uznXMzJkz9fbbb+v111/XBx98oAMHDmjcuHG+/ubM2QMA0BY2bNjg9nnlypXq1q2biouLddVVV+nw4cN68cUXtWbNGl199dWSpBUrVqh///766KOPdOmll/osFip7AIA5+GgYv6amxm1raGho0eUPHz4sSYqJiZEkFRcXq6mpSampqa5j+vXrp8TERBUVFfn0q5PsAQAm4W2ib072CQkJioqKcm05OTm/eGWn06kZM2bo8ssv1wUXXCBJqqioUFhYmKKjo92OjY2NVUVFhU+/OcP4AAB4oLy8XHa73fXZZrP9Yp+pU6fq888/19atW9sytNMi2QMAzMFHT9Cz2+1uyf6XZGZmav369SosLNS5557rao+Li1NjY6Oqq6vdqvvKykrFxcW1Ps5TYBgfAGAO7bwa3zAMZWZmau3atdq0aZOSkpLc9g8dOlQdOnRQQUGBq620tFT79u1TSkqKT77yCVT2AAC0galTp2rNmjX67//+b0VGRrrm4aOiohQREaGoqChNmjRJWVlZiomJkd1u17Rp05SSkuLTlfgSyR4AYBaGs3nzpr8Hli9fLkkaMWKEW/uKFSt02223SZKWLFkiq9Wq8ePHq6GhQWlpaXrmmWdaH+NpkOwBAObQzm+9M1pwfHh4uPLy8pSXl9faqFqEZA8AMAfnj7fPtb5/YGKBHgAAQY7KHgBgDu08jH8mIdkDAMzBkJfJ3meRtDuG8QEACHJU9gAAc2AYHwCAIOd0SvLiPnunF339jGF8AACCHJU9AMAcGMYHACDImTjZM4wPAECQo7IHAJiDiR+XS7IHAJiCYThlePHWO2/6+hvJHgBgDobhXXXOnD0AADhTUdkDAMzB8HLOPoAre5I9AMAcnE7J4sW8ewDP2TOMDwBAkKOyBwCYA8P4AAAEN8PplOHFMH4g33rHMD4AAEGOyh4AYA4M4wMAEOSchmQxZ7JnGB8AgCBHZQ8AMAfDkOTNffaBW9mT7AEApmA4DRleDOMbJHsAAM5whlPeVfbcegcAAM5QVPYAAFNgGB8AgGBn4mH8gE72J/7KOq4mr56TAJzJao4E7i8Y4JfU1Db/fLdH1extrjiuJt8F084COtkfOXJEkrRV7/o5EqDtdOnj7wiAtnfkyBFFRUW1ybnDwsIUFxenrRXe54q4uDiFhYX5IKr2ZTECeBLC6XTqwIEDioyMlMVi8Xc4plBTU6OEhASVl5fLbrf7OxzAp/j5bn+GYejIkSOKj4+X1dp2a8br6+vV2Njo9XnCwsIUHh7ug4jaV0BX9larVeeee66/wzAlu93OL0MELX6+21dbVfT/U3h4eEAmaV/h1jsAAIIcyR4AgCBHsodHbDab7r//ftlsNn+HAvgcP98IVgG9QA8AAPwyKnsAAIIcyR4AgCBHsgcAIMiR7NFieXl56tmzp8LDw5WcnKyPP/7Y3yEBPlFYWKjRo0crPj5eFotF69at83dIgE+R7NEir776qrKysnT//fdr586dGjRokNLS0nTo0CF/hwZ4ra6uToMGDVJeXp6/QwHaBKvx0SLJycm6+OKL9fTTT0tqflRxQkKCpk2bprlz5/o5OsB3LBaL1q5dq7Fjx/o7FMBnqOzxixobG1VcXKzU1FRXm9VqVWpqqoqKivwYGQCgJUj2+EXff/+9HA6HYmNj3dpjY2NVUVHhp6gAAC1FsgcAIMiR7PGLzjrrLIWEhKiystKtvbKyUnFxcX6KCgDQUiR7/KKwsDANHTpUBQUFrjan06mCggKlpKT4MTIAQEsE9Pvs0X6ysrKUkZGhYcOG6ZJLLlFubq7q6uo0ceJEf4cGeK22tlZ79uxxfS4rK9OuXbsUExOjxMREP0YG+Aa33qHFnn76aT322GOqqKjQ4MGDtXTpUiUnJ/s7LMBrmzdv1siRI09qz8jI0MqVK9s/IMDHSPYAAAQ55uwBAAhyJHsAAIIcyR4AgCBHsgcAIMiR7AEACHIkewAAghzJHgCAIEeyBwAgyJHsAS/ddtttGjt2rOvziBEjNGPGjHaPY/PmzbJYLKqurj7tMRaLRevWrWvxORcsWKDBgwd7Fdc333wji8WiXbt2eXUeAK1HskdQuu2222SxWGSxWBQWFqZevXpp0aJFOn78eJtf+80339QDDzzQomNbkqABwFu8CAdB67rrrtOKFSvU0NCgd999V1OnTlWHDh00b968k45tbGxUWFiYT64bExPjk/MAgK9Q2SNo2Ww2xcXFqUePHvrjH/+o1NRUvfXWW5J+HHp/6KGHFB8fr759+0qSysvLddNNNyk6OloxMTEaM2aMvvnmG9c5HQ6HsrKyFB0dra5du2r27Nn66eslfjqM39DQoDlz5ighIUE2m029evXSiy++qG+++cb18pUuXbrIYrHotttuk9T8CuGcnBwlJSUpIiJCgwYN0l//+le367z77rvq06ePIiIiNHLkSLc4W2rOnDnq06ePOnbsqPPOO0/Z2dlqamo66bhnn31WCQkJ6tixo2666SYdPnzYbf8LL7yg/v37Kzw8XP369dMzzzzjcSwA2g7JHqYRERGhxsZG1+eCggKVlpZq48aNWr9+vZqampSWlqbIyEht2bJFH374oTp37qzrrrvO1e+JJ57QypUr9dJLL2nr1q2qqqrS2rVrf/a6f/jDH/SXv/xFS5cuVUlJiZ599ll17txZCQkJeuONNyRJpaWlOnjwoJ566ilJUk5OjlatWqX8/Hzt3r1bM2fO1C233KIPPvhAUvMfJePGjdPo0aO1a9cu3XHHHZo7d67H/yeRkZFauXKl/vnPf+qpp57S888/ryVLlrgds2fPHr322mt6++23tWHDBn366ae66667XPtXr16t+fPn66GHHlJJSYkefvhhZWdn6+WXX/Y4HgBtxACCUEZGhjFmzBjDMAzD6XQaGzduNGw2m3Hvvfe69sfGxhoNDQ2uPn/+85+Nvn37Gk6n09XW0NBgREREGO+9955hGIbRvXt3Y/Hixa79TU1Nxrnnnuu6lmEYxvDhw427777bMAzDKC0tNSQZGzduPGWc77//viHJ+OGHH1xt9fX1RseOHY1t27a5HTtp0iTj5ptvNgzDMObNm2cMGDDAbf+cOXNOOtdPSTLWrl172v2PPfaYMXToUNfn+++/3wgJCTG+++47V9vf/vY3w2q1GgcPHjQMwzB+9atfGWvWrHE7zwMPPGCkpKQYhmEYZWVlhiTj008/Pe11AbQt5uwRtNavX6/OnTurqalJTqdT//mf/6kFCxa49g8cONBtnv6zzz7Tnj17FBkZ6Xae+vp67d27V4cPH9bBgweVnJzs2hcaGqphw4adNJR/wq5duxQSEqLhw4e3OO49e/bo6NGj+vWvf+3W3tjYqCFDhkiSSkpK3OKQpJSUlBZf44RXX31VS5cu1d69e1VbW6vjx4/Lbre7HZOYmKhzzjnH7TpOp1OlpaWKjIzU3r17NWnSJE2ePNl1zPHjxxUVFeVxPADaBskeQWvkyJFavny5wsLCFB8fr9BQ9x/3Tp06uX2ura3V0KFDtXr16pPOdfbZZ7cqhoiICI/71NbWSpLeeecdtyQrNa9D8JWioiKlp6dr4cKFSktLU1RUlF555RU98cQTHsf6/PPPn/THR0hIiM9iBeAdkj2CVqdOndSrV68WH3/RRRfp1VdfVbdu3U6qbk/o3r27tm/frquuukpScwVbXFysiy666JTHDxw4UE6nUx988IFSU1NP2n9iZMHhcLjaBgwYIJvNpn379p12RKB///6uxYYnfPTRR7/8Jf+Hbdu2qUePHvrTn/7kavv2229POm7fvn06cOCA4uPjXdexWq3q27evYmNjFR8fr6+//lrp6ekeXR9A+2GBHvBv6enpOuusszRmzBht2bJFZWVl2rx5s6ZPn67vvvtOknT33XfrkUce0bp16/TFF1/orrvu+tl75Hv27KmMjAzdfvvtWrduneucr732miSpR48eslgsWr9+vf71r3+ptrZWkZGRuvfeezVz5ky9/PLL2rt3r3bu3Klly5a5Fr3deeed+uqrrzRr1iyVlpZqzZo1WrlypUfft3fv3tq3b59eeeUV7d27V0uXLj3lYsPw8HBlZGTos88+05YtWzR9+nTddNNNiouLkyQtXLhQOTk5Wrp0qb788kv94x//0IoVK/Tkk096FA+AtkOyB/6tY8eOKiwsVGJiosaNG6f+/ftr0qRJqq+vd1X699xzj2699VZlZGQoJSVFkZGR+u1vf/uz512+fLl+97vf6a677lK/fv00efJk1dXVSZLOOeccLVy4UHPnzlVsbKwyMzMlSQ888ICys7OVk5Oj/v3767rrrtM777yjpKQkSc3z6G+88YbWrVunQYMGKT8/Xw8//LBH3/fGG2/UzJkzlZmZqcGDB2vbtm3Kzs4+6bhevXpp3Lhxuv7663XttdfqwgsvdLu17o477tALL7ygFStWaODAgRo+fLhWrlzpihWA/1mM060sAgAAQYHKHgCAIEeyBwAgyJHsAQAIciR7AACCHMkeAIAgR7IHACDIkewBAAhyJHsAAIIcyR4AgCBHsgcAIMiR7AEACHIkewAAgtz/BytbhoiuNtZdAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import ConfusionMatrixDisplay\n",
    "\n",
    "ConfusionMatrixDisplay.from_estimator(estimator=clf, X=X, y=y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAGwCAYAAABSAee3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArNElEQVR4nO3deXQUdbr/8U8nIZ1AFohAQiAEMLLJ5qAyuCCMDIv3Kojzc1S8NyDiVQkqiCwqIKBkRq+CKAOOCxF/cNWrwgjj4A9BNlm8oOjgQMZEEBCCC0JIMAtd9fsj0t42IN2p7nRX1/t1Tp2T/qaWJ5wcnjzP91tVLtM0TQEAAFuKCXcAAACg7kjkAADYGIkcAAAbI5EDAGBjJHIAAGyMRA4AgI2RyAEAsLG4cAdghWEYOnTokJKTk+VyucIdDgAgQKZp6sSJE8rMzFRMTOhqy4qKClVVVVk+T3x8vBISEoIQUfDYOpEfOnRIWVlZ4Q4DAGDRgQMH1KpVq5Ccu6KiQm2zk1TytcfyuTIyMrR3796ISua2TuTJycmSpEfWXq6EJFv/KMBZre7PH6uIXqfMKq0/8br3//NQqKqqUsnXHn25o41Skute9ZeeMJTdc5+qqqpI5MFyup2ekBRHIkfUinPFhzsEIOTqY3o0KdmlpOS6X8dQZE7hkv0AAI7gMQ15LLxdxGMawQsmiEjkAABHMGTKUN0zuZVjQ4nbzwAAsDEqcgCAIxgyZKU5bu3o0CGRAwAcwWOa8ph1b49bOTaUaK0DAGBjVOQAAEeI1sVuJHIAgCMYMuWJwkROax0AABujIgcAOAKtdQAAbIxV6wAAIOJQkQMAHMH4cbNyfCQikQMAHMFjcdW6lWNDiUQOAHAEjymLbz8LXizBxBw5AAA2RkUOAHAE5sgBALAxQy555LJ0fCSitQ4AgI1RkQMAHMEwazYrx0ciEjkAwBE8FlvrVo4NJVrrAADYGBU5AMARorUiJ5EDABzBMF0yTAur1i0cG0q01gEAsDEqcgCAI9BaBwDAxjyKkcdCI9oTxFiCiUQOAHAE0+IcuckcOQAACDYqcgCAIzBHDgCAjXnMGHlMC3PkEfqIVlrrAADYGBU5AMARDLlkWKhfDUVmSU4iBwA4QrTOkdNaBwDAxqjIAQCOYH2xG611AADCpmaO3MJLU2itAwCAYKMiBwA4gmHxWeusWgcAIIyidY6c1joAwBEMxVjeApGfn69LLrlEycnJat68uYYOHarCwkKfffr27SuXy+Wz3XnnnQFdh0QOAEAIrF+/XmPGjNHWrVu1evVqVVdXa8CAASovL/fZb/To0Tp8+LB3e/zxxwO6Dq11AIAjeEyXPBZeRXr62NLSUp9xt9stt9tda/9Vq1b5fC4oKFDz5s21Y8cO9enTxzvesGFDZWRk1DkuKnIAgCN4flzsZmWTpKysLKWmpnq3/Px8v65//PhxSVJaWprP+JIlS9S0aVN16dJFU6ZM0cmTJwP6uajIAQAIwIEDB5SSkuL9fKZq/OcMw9B9992nyy+/XF26dPGO33LLLcrOzlZmZqY+/fRTTZo0SYWFhXrrrbf8jodEDgBwBMOMkWFh1brx46r1lJQUn0TujzFjxmjXrl3atGmTz/gdd9zh/bpr165q0aKFrr76ahUXF+v888/369y01gEAjhCs1nqg8vLytHLlSr3//vtq1arVL+7bq1cvSVJRUZHf56ciBwAgBEzT1NixY7Vs2TKtW7dObdu2PecxO3fulCS1aNHC7+uQyAEAjmBIllatGwHuP2bMGC1dulR/+ctflJycrJKSEklSamqqEhMTVVxcrKVLl+qaa67Reeedp08//VTjxo1Tnz591K1bN7+vQyIHADhCXR7q8vPjA7FgwQJJNQ99+d8WLVqkESNGKD4+Xu+9957mzp2r8vJyZWVl6YYbbtDDDz8c0HVI5AAAhIB5jke6ZmVlaf369ZavQyIHADiC9WetR+b6cBI5AMARovV95CRyAIAjRGtFHplRAQAAv1CRAwAcwcpDXU4fH4lI5AAARzBMlwwr95FbODaUIvPPCwAA4BcqcgCAIxgWW+tWHiYTSiRyAIAjWH/7WWQm8siMCgAA+IWKHADgCB655LHwUBcrx4YSiRwA4Ai01gEAQMShIgcAOIJH1trjnuCFElQkcgCAI0Rra51EDgBwBF6aAgAAIg4VOQDAEUyL7yM3uf0MAIDwobUOAAAiDhU5AMARovU1piRyAIAjeCy+/czKsaEUmVEBAAC/UJEDAByB1joAADZmKEaGhUa0lWNDKTKjAgAAfqEiBwA4gsd0yWOhPW7l2FAikQMAHIE5cgAAbMy0+PYzkye7AQCAYKMiBwA4gkcueSy8+MTKsaFEIgcAOIJhWpvnNswgBhNEtNYBALAxKnLUUvR8gg6vbqCyvbGKTTDVpMcpdRr/g5LaGrX2NU3pwzuT9M2mBrp4Xpkyrq4OQ8SANcPzvtTwvP0+Ywe+SNR/XHNxmCJCKBgWF7tZOTaUSOSo5bv/iVObmyvVuOspmadc2vN0oraNTtJVb5cqrqHvvnsXuxWh00ZAQPb9s6Eeuq2r97PnFL/Y0caQS4aF/7CsHBtKEfHnxfz589WmTRslJCSoV69e+vDDD8MdkqP1+nOZsq6vUnKOoZSOHnV/rFw/HI7V8X/4/t13fHesvng5Qd1nlYcpUiB4PB6Xvv823ruVHmsQ7pAAv4Q9kb/22msaP368pk+fro8++kjdu3fXwIED9fXXX4c7NPzo1Imav0IbpP7UWvf8IH08sZG6PHxSCc0idAUIEICW2T/olQ3b9OLq/9EDT+xRsxYV4Q4JQXb6yW5WtkgU9kT+1FNPafTo0Ro5cqQ6d+6shQsXqmHDhnrppZfCHRokmYb02R8T1eSiU0q54KdE/tkfG6rJRaeU8RvmxGF/hZ8k66kp7TX19i6aPyNH6a0q9MT//VSJjU6FOzQE0ek5citbJArrHHlVVZV27NihKVOmeMdiYmLUv39/bdmypdb+lZWVqqys9H4uLS2tlzidbNejDXXi81hd9soJ71jJ2gb6dluc+rzBvz+iw/aNad6v9/2zkQo/SVbB2g915aBv9f/ezAhjZMC5hTWRf/vtt/J4PEpPT/cZT09P1549e2rtn5+frxkzZtRXeI7390cTdWR9A1328gklZvzUPv9uW5xOHojRu70b++y//b5GSut5SpcVlNVzpEBwlZ+I01f7EpWZ/UO4Q0EQGbL4rPUIXexmq1XrU6ZM0fjx472fS0tLlZWVFcaIopNpSrseS1TJmnj1Ljihhq18bzs7//YKZf2u0mdsw9BUXTjpB6X3pdUO+0to6FGLrAqtfTs+3KEgiEyLq9ZNEnltTZs2VWxsrI4cOeIzfuTIEWVk1G5nud1uud3u+grPsXbNStRX78TrkmfKFdfQVMU3Py52SzYVmyAlNDPPuMAtsYVRK+kDdjBq4hfa9n6avj6UoPOaV+nWvC9lGNK6lc3CHRqCiLefhUB8fLx69uypNWvWaOjQoZIkwzC0Zs0a5eXlhTM0R/vytQRJ0pYRyT7j3R8tV9b1VeEICQippumVmvRkoVIaV+v40Qb6bEeKxv2+h0q/pyJH5At7a338+PHKzc3VxRdfrEsvvVRz585VeXm5Ro4cGe7QHOtfP/u+Xo4BIsUf7+8U7hBQD3iyW4j8/ve/1zfffKNp06appKREPXr00KpVq2otgAMAwApa6yGUl5dHKx0AgDqIiEQOAECoReuz1knkAABHiNbWemTO3AMAAL9QkQMAHCFaK3ISOQDAEaI1kdNaBwDAxqjIAQCOEK0VOYkcAOAIpqzdQlb7DRORgUQOAHCEaK3ImSMHAMDGqMgBAI4QrRU5iRwA4AjRmshprQMAYGNU5AAAR4jWipxEDgBwBNN0ybSQjK0cG0q01gEAsDEqcgCAI/A+cgAAbCxa58hprQMAYGNU5AAAR2CxGwAANna6tW5lC0R+fr4uueQSJScnq3nz5ho6dKgKCwt99qmoqNCYMWN03nnnKSkpSTfccIOOHDkS0HVI5AAARzhdkVvZArF+/XqNGTNGW7du1erVq1VdXa0BAwaovLzcu8+4ceO0YsUK/fd//7fWr1+vQ4cOadiwYQFdh9Y6AAABKC0t9fnsdrvldrtr7bdq1SqfzwUFBWrevLl27NihPn366Pjx43rxxRe1dOlS/eY3v5EkLVq0SJ06ddLWrVv161//2q94qMgBAI5gWmyrn67Is7KylJqa6t3y8/P9uv7x48clSWlpaZKkHTt2qLq6Wv379/fu07FjR7Vu3Vpbtmzx++eiIgcAOIIpyTStHS9JBw4cUEpKinf8TNX4zxmGofvuu0+XX365unTpIkkqKSlRfHy8Gjdu7LNvenq6SkpK/I6LRA4AQABSUlJ8Erk/xowZo127dmnTpk1Bj4dEDgBwBEMuucLwZLe8vDytXLlSGzZsUKtWrbzjGRkZqqqq0rFjx3yq8iNHjigjI8Pv8zNHDgBwhPpetW6apvLy8rRs2TKtXbtWbdu29fl+z5491aBBA61Zs8Y7VlhYqP3796t3795+X4eKHACAEBgzZoyWLl2qv/zlL0pOTvbOe6empioxMVGpqakaNWqUxo8fr7S0NKWkpGjs2LHq3bu33yvWJRI5AMAhDNMlVz0+a33BggWSpL59+/qML1q0SCNGjJAkzZkzRzExMbrhhhtUWVmpgQMH6k9/+lNA1yGRAwAcwTQtrloP8FjTjwMSEhI0f/58zZ8/v45RMUcOAICtUZEDABwhWl+aQiIHADgCiRwAABur78Vu9YU5cgAAbIyKHADgCPW9ar2+kMgBAI5Qk8itzJEHMZggorUOAICNUZEDAByBVesAANiYqZ/eKV7X4yMRrXUAAGyMihwA4Ai01gEAsLMo7a2TyAEAzmCxIleEVuTMkQMAYGNU5AAAR+DJbgAA2Fi0LnajtQ4AgI1RkQMAnMF0WVuwFqEVOYkcAOAI0TpHTmsdAAAboyIHADgDD4QBAMC+onXVul+J/O233/b7hNddd12dgwEAAIHxK5EPHTrUr5O5XC55PB4r8QAAEDoR2h63wq9EbhhGqOMAACCkorW1bmnVekVFRbDiAAAgtMwgbBEo4ETu8Xg0a9YstWzZUklJSfriiy8kSVOnTtWLL74Y9AABAMDZBZzIH3vsMRUUFOjxxx9XfHy8d7xLly564YUXghocAADB4wrCFnkCTuSLFy/Wn//8Zw0fPlyxsbHe8e7du2vPnj1BDQ4AgKChtV7jq6++Uk5OTq1xwzBUXV0dlKAAAIB/Ak7knTt31saNG2uNv/HGG7rooouCEhQAAEEXpRV5wE92mzZtmnJzc/XVV1/JMAy99dZbKiws1OLFi7Vy5cpQxAgAgHVR+vazgCvyIUOGaMWKFXrvvffUqFEjTZs2Tbt379aKFSv029/+NhQxAgCAs6jTs9avvPJKrV69OtixAAAQMtH6GtM6vzRl+/bt2r17t6SaefOePXsGLSgAAIKOt5/VOHjwoG6++WZ98MEHaty4sSTp2LFjuuyyy/Tqq6+qVatWwY4RAACcRcBz5Lfffruqq6u1e/duHT16VEePHtXu3btlGIZuv/32UMQIAIB1pxe7WdkiUMAV+fr167V582Z16NDBO9ahQwc988wzuvLKK4MaHAAAweIyazYrx0eigBN5VlbWGR/84vF4lJmZGZSgAAAIuiidIw+4tf7EE09o7Nix2r59u3ds+/btuvfee/Wf//mfQQ0OAAD8Mr8q8iZNmsjl+mluoLy8XL169VJcXM3hp06dUlxcnG677TYNHTo0JIECAGBJlD4Qxq9EPnfu3BCHAQBAiEVpa92vRJ6bmxvqOAAAQB3U+YEwklRRUaGqqiqfsZSUFEsBAQAQElFakQe82K28vFx5eXlq3ry5GjVqpCZNmvhsAABEpCh9+1nAiXzixIlau3atFixYILfbrRdeeEEzZsxQZmamFi9eHIoYAQDAWQTcWl+xYoUWL16svn37auTIkbryyiuVk5Oj7OxsLVmyRMOHDw9FnAAAWBOlq9YDrsiPHj2qdu3aSaqZDz969Kgk6YorrtCGDRuCGx0AAEFy+sluVrZIFHAib9eunfbu3StJ6tixo15//XVJNZX66ZeoAACA+hFwIh85cqQ++eQTSdLkyZM1f/58JSQkaNy4cXrggQeCHiAAAEERpYvdAp4jHzdunPfr/v37a8+ePdqxY4dycnLUrVu3oAYHAAB+maX7yCUpOztb2dnZwYgFAICQccni28+CFklw+ZXI582b5/cJ77nnnjoHAwAAAuNXIp8zZ45fJ3O5XGFJ5Ksubaw4V4N6vy5QH949xN0giF6lJww1aV9PF4vS28/8SuSnV6kDAGBbPKIVAABEGsuL3QAAsIUorchJ5AAAR7D6dLaoebIbAACIHFTkAABniNLWep0q8o0bN+rWW29V79699dVXX0mSXnnlFW3atCmowQEAEDRR+ojWgBP5m2++qYEDByoxMVEff/yxKisrJUnHjx/X7Nmzgx4gAAA4u4AT+aOPPqqFCxfq+eefV4MGPz2E5fLLL9dHH30U1OAAAAiWaH2NacBz5IWFherTp0+t8dTUVB07diwYMQEAEHxR+mS3gCvyjIwMFRUV1RrftGmT2rVrF5SgAAAIOubIa4wePVr33nuvtm3bJpfLpUOHDmnJkiWaMGGC7rrrrlDECACA7WzYsEHXXnutMjMz5XK5tHz5cp/vjxgxQi6Xy2cbNGhQwNcJuLU+efJkGYahq6++WidPnlSfPn3kdrs1YcIEjR07NuAAAACoD/X9QJjy8nJ1795dt912m4YNG3bGfQYNGqRFixZ5P7vd7oDjCjiRu1wuPfTQQ3rggQdUVFSksrIyde7cWUlJSQFfHACAehOk+8hLS0t9ht1u9xkT8ODBgzV48OBfPKXb7VZGRoaFoCw82S0+Pl6dO3fWpZdeShIHADhGVlaWUlNTvVt+fn6dz7Vu3To1b95cHTp00F133aXvvvsu4HMEXJH369dPLtfZV+6tXbs24CAAAAg5q7eQ/XjsgQMHlJKS4h2uSztcqmmrDxs2TG3btlVxcbEefPBBDR48WFu2bFFsbKzf5wk4kffo0cPnc3V1tXbu3Kldu3YpNzc30NMBAFA/gtRaT0lJ8UnkdXXTTTd5v+7atau6deum888/X+vWrdPVV1/t93kCTuRz5sw54/gjjzyisrKyQE8HAAAktWvXTk2bNlVRUVFAiTxobz+79dZb9dJLLwXrdAAABFeE30d+8OBBfffdd2rRokVAxwXt7WdbtmxRQkJCsE4HAEBQ1fftZ2VlZT4PUNu7d6927typtLQ0paWlacaMGbrhhhuUkZGh4uJiTZw4UTk5ORo4cGBA1wk4kf/8XjjTNHX48GFt375dU6dODfR0AABEpe3bt6tfv37ez+PHj5ck5ebmasGCBfr000/18ssv69ixY8rMzNSAAQM0a9asgBfPBZzIU1NTfT7HxMSoQ4cOmjlzpgYMGBDo6QAAiEp9+/aVaZ69jH/33XeDcp2AErnH49HIkSPVtWtXNWnSJCgBAABQL4K0aj3SBLTYLTY2VgMGDOAtZwAA24nW15gGvGq9S5cu+uKLL0IRCwAACFDAifzRRx/VhAkTtHLlSh0+fFilpaU+GwAAEStCbz2zwu858pkzZ+r+++/XNddcI0m67rrrfB7VapqmXC6XPB5P8KMEAMCqKJ0j9zuRz5gxQ3feeafef//9UMYDAAAC4HciP72E/qqrrgpZMAAAhEp9PxCmvgR0+9kvvfUMAICI5vTWuiS1b9/+nMn86NGjlgICAAD+CyiRz5gxo9aT3QAAsANa66p5d2rz5s1DFQsAAKETpa11v+8jZ34cAIDIE/CqdQAAbClKK3K/E7lhGKGMAwCAkGKOHAAAO4vSijzgZ60DAIDIQUUOAHCGKK3ISeQAAEeI1jlyWusAANgYFTkAwBlorQMAYF+01gEAQMShIgcAOAOtdQAAbCxKEzmtdQAAbIyKHADgCK4fNyvHRyISOQDAGaK0tU4iBwA4ArefAQCAiENFDgBwBlrrAADYXIQmYytorQMAYGNU5AAAR4jWxW4kcgCAM0TpHDmtdQAAbIyKHADgCLTWAQCwM1rrAAAg0lCRAwAcgdY6AAB2FqWtdRI5AMAZojSRM0cOAICNUZEDAByBOXIAAOyM1joAAIg0VOQAAEdwmaZcZt3LaivHhhKJHADgDLTWAQBApKEiBwA4AqvWAQCwM1rrAAAg0lCRAwAcgdY6AAB2FqWtdRI5AMARorUiZ44cAAAboyIHADgDrXUAAOwtUtvjVtBaBwDAxqjIAQDOYJo1m5XjIxCJHADgCKxaBwAAEYeKHADgDKxaBwDAvlxGzWbl+EhEax0AABujIkfAbsw7olEPlmjZ8021cHrLcIcDBOzVZ5rrg3ca60CRW/EJhjpffFKjHjqkrJxK7z5Hv47TC7My9dGGZJ0si1HW+ZW66d4juvJfjocxclgSpa11KnIEpH33k/qXW4/qi88Swh0KUGefbknStSO+1dyVnyv/1WJ5TkkP3ny+Kk7+9F/iE/e01oFitx4p2Kvn1hbq8muOa/Z/tFHR3xPDGDmsOL1q3coWicKayDds2KBrr71WmZmZcrlcWr58eTjDwTkkNPRo0rNfau4DrXTieGy4wwHqbPbSLzTg90fVpkOFzr+wQvfP3a+vv4rX55/+lKT/sb2Rhtz2rTpedFItsqt0y31H1CjV47MPbOb0feRWtgCcK8eZpqlp06apRYsWSkxMVP/+/fX5558H/GOFNZGXl5ere/fumj9/fjjDgJ/yZn+lD9ek6OONyeEOBQiq8tKaP0yTG3u8Y50vLtf6txur9PtYGYa0bnljVVW41O2ysnCFCZs5V457/PHHNW/ePC1cuFDbtm1To0aNNHDgQFVUVAR0nbDOkQ8ePFiDBw/2e//KykpVVv40h1VaWhqKsHAGVw35Xjldf9DYay4IdyhAUBmGtHB6S114SZnadPzpP9CHnvtSs+/M1v+5sKti40y5Ew1Nf3GfWratCmO0sCJYD4T5ee5xu91yu9219v+lHGeapubOnauHH35YQ4YMkSQtXrxY6enpWr58uW666Sa/47LVHHl+fr5SU1O9W1ZWVrhDcoRmmVW6a+Yh/TGvtaorbfUrA5zTsw+20pd7EjVlwZc+4y8/nqGy0lj94bUiPfO3Qt1wx9d67M422rub9SG2ZQZhk5SVleWTi/Lz8wMOZe/evSopKVH//v29Y6mpqerVq5e2bNkS0LlstWp9ypQpGj9+vPdzaWkpybwe5HT7QU2andL8d//pHYuNk7r+ulzXjfxW/9qmmwzDFcYIgbp59sGW2rY6RU8uK1KzzGrv+KF98Xp7UTM99/4etelQU6Wff2GF/r4tSW8XNNW9fzwYrpARAQ4cOKCUlBTv5zNV4+dSUlIiSUpPT/cZT09P937PX7ZK5GdrXyC0dm5M0h392vuM3T/ngA4UJej1+c1I4rAd05TmP9RSm1el6ok3ipTR2rddXvlDTecpJsa3Dxsba8qM0IeC4NyC1VpPSUnxSeThRp8U5/RDeay+LEz02SpOxujE9zXjgN08+2ArrX0rTZPnf6nEJENHv47T0a/jVPlDzR+lWTkVymxbqacnZmnPxw11aF+83ljYTB9tSNZlg7iP3LbqedX6L8nIyJAkHTlyxGf8yJEj3u/5y1YVOQAEw8qXm0qSHrjBd/Hm/XP2a8DvjyqugfToK8V6cXampue21Q/lMcpsW6UJT+/XpVefCEfIiDJt27ZVRkaG1qxZox49ekiqmS7etm2b7rrrroDOFdZEXlZWpqKiIu/nvXv3aufOnUpLS1Pr1q3DGBnOZeLvcsIdAlBn7x7aec59Wrar0rQX9oU8FtSf+n6N6bly3H333adHH31UF1xwgdq2baupU6cqMzNTQ4cODeg6YU3k27dvV79+/byfTy9ky83NVUFBQZiiAgBEpXp+ROu5ctzEiRNVXl6uO+64Q8eOHdMVV1yhVatWKSEhsDsjwprI+/btKzOIcw4AAESKc+U4l8ulmTNnaubMmZauwxw5AMAR6ru1Xl9I5AAAZzDMms3K8RGIRA4AcAZeYwoAACINFTkAwBFcsjhHHrRIgotEDgBwBqtPZ4vQu6xorQMAYGNU5AAAR+D2MwAA7IxV6wAAINJQkQMAHMFlmnJZWLBm5dhQIpEDAJzB+HGzcnwEorUOAICNUZEDAByB1joAAHYWpavWSeQAAGfgyW4AACDSUJEDAByBJ7sBAGBntNYBAECkoSIHADiCy6jZrBwfiUjkAABnoLUOAAAiDRU5AMAZeCAMAAD2Fa2PaKW1DgCAjVGRAwCcIUoXu5HIAQDOYMraO8UjM4+TyAEAzsAcOQAAiDhU5AAAZzBlcY48aJEEFYkcAOAMUbrYjdY6AAA2RkUOAHAGQ5LL4vERiEQOAHAEVq0DAICIQ0UOAHCGKF3sRiIHADhDlCZyWusAANgYFTkAwBmitCInkQMAnIHbzwAAsC9uPwMAABGHihwA4AzMkQMAYGOGKbksJGMjMhM5rXUAAGyMihwA4Ay01gEAsDOLiVyRmchprQMAYGNU5AAAZ6C1DgCAjRmmLLXHWbUOAACCjYocAOAMplGzWTk+ApHIAQDOwBw5AAA2xhw5AACINFTkAABnoLUOAICNmbKYyIMWSVDRWgcAwMaoyAEAzkBrHQAAGzMMSRbuBTci8z5yWusAANgYFTkAwBlorQMAYGNRmshprQMAYGNU5AAAZ+ARrQAA2JdpGpa3QDzyyCNyuVw+W8eOHYP+c1GRAwCcwTStVdV1mCO/8MIL9d5773k/x8UFP+2SyAEACEBpaanPZ7fbLbfbfcZ94+LilJGREdJ4aK0DAJzh9Kp1K5ukrKwspaamerf8/PyzXvLzzz9XZmam2rVrp+HDh2v//v1B/7GoyAEAzmAYksvC09l+nCM/cOCAUlJSvMNnq8Z79eqlgoICdejQQYcPH9aMGTN05ZVXateuXUpOTq57HD9DIgcAIAApKSk+ifxsBg8e7P26W7du6tWrl7Kzs/X6669r1KhRQYuHRA4AcAbT4u1nFh8I07hxY7Vv315FRUWWzvNzzJEDABzBNAzLmxVlZWUqLi5WixYtgvQT1SCRAwAQAhMmTND69eu1b98+bd68Wddff71iY2N18803B/U6tNYBAM5Qz631gwcP6uabb9Z3332nZs2a6YorrtDWrVvVrFmzusdwBiRyAIAzGKbkqr9E/uqrr9b9WgGgtQ4AgI1RkQMAnME0JVm5jzwyX5pCIgcAOIJpmDIttNZNEjkAAGFkGrJWkVu7/SxUmCMHAMDGqMgBAI5Aax0AADuL0ta6rRP56b+OTqna0j3+QCQrPRGZ/3kAwVBaVvP7XR/VrtVccUrVwQsmiGydyE+cOCFJ2qR3whwJEDpN2oc7AiD0Tpw4odTU1JCcOz4+XhkZGdpUYj1XZGRkKD4+PghRBY/LjNSmvx8Mw9ChQ4eUnJwsl8sV7nAcobS0VFlZWbXexwtEA36/659pmjpx4oQyMzMVExO69dcVFRWqqqqyfJ74+HglJCQEIaLgsXVFHhMTo1atWoU7DEfy9328gB3x+12/QlWJ/28JCQkRl4CDhdvPAACwMRI5AAA2RiJHQNxut6ZPny632x3uUICg4/cbdmTrxW4AADgdFTkAADZGIgcAwMZI5AAA2BiJHAAAGyORw2/z589XmzZtlJCQoF69eunDDz8Md0hAUGzYsEHXXnutMjMz5XK5tHz58nCHBPiNRA6/vPbaaxo/frymT5+ujz76SN27d9fAgQP19ddfhzs0wLLy8nJ1795d8+fPD3coQMC4/Qx+6dWrly655BI9++yzkmqec5+VlaWxY8dq8uTJYY4OCB6Xy6Vly5Zp6NCh4Q4F8AsVOc6pqqpKO3bsUP/+/b1jMTEx6t+/v7Zs2RLGyAAAJHKc07fffiuPx6P09HSf8fT0dJWUlIQpKgCARCIHAMDWSOQ4p6ZNmyo2NlZHjhzxGT9y5IgyMjLCFBUAQCKRww/x8fHq2bOn1qxZ4x0zDENr1qxR7969wxgZACAu3AHAHsaPH6/c3FxdfPHFuvTSSzV37lyVl5dr5MiR4Q4NsKysrExFRUXez3v37tXOnTuVlpam1q1bhzEy4Ny4/Qx+e/bZZ/XEE0+opKREPXr00Lx589SrV69whwVYtm7dOvXr16/WeG5urgoKCuo/ICAAJHIAAGyMOXIAAGyMRA4AgI2RyAEAsDESOQAANkYiBwDAxkjkAADYGIkcAAAbI5EDAGBjJHLAohEjRmjo0KHez3379tV9991X73GsW7dOLpdLx44dO+s+LpdLy5cv9/ucjzzyiHr06GEprn379snlcmnnzp2WzgPgzEjkiEojRoyQy+WSy+VSfHy8cnJyNHPmTJ06dSrk137rrbc0a9Ysv/b1J/kCwC/hpSmIWoMGDdKiRYtUWVmpd955R2PGjFGDBg00ZcqUWvtWVVUpPj4+KNdNS0sLynkAwB9U5IhabrdbGRkZys7O1l133aX+/fvr7bfflvRTO/yxxx5TZmamOnToIEk6cOCAbrzxRjVu3FhpaWkaMmSI9u3b5z2nx+PR+PHj1bhxY5133nmaOHGifv66gp+31isrKzVp0iRlZWXJ7XYrJydHL774ovbt2+d9UUeTJk3kcrk0YsQISTWvic3Pz1fbtm2VmJio7t2764033vC5zjvvvKP27dsrMTFR/fr184nTX5MmTVL79u3VsGFDtWvXTlOnTlV1dXWt/Z577jllZWWpYcOGuvHGG3X8+HGf77/wwgvq1KmTEhIS1LFjR/3pT38KOBYAdUMih2MkJiaqqqrK+3nNmjUqLCzU6tWrtXLlSlVXV2vgwIFKTk7Wxo0b9cEHHygpKUmDBg3yHvfkk0+qoKBAL730kjZt2qSjR49q2bJlv3jdf//3f9d//dd/ad68edq9e7eee+45JSUlKSsrS2+++aYkqbCwUIcPH9bTTz8tScrPz9fixYu1cOFCffbZZxo3bpxuvfVWrV+/XlLNHxzDhg3Ttddeq507d+r222/X5MmTA/43SU5OVkFBgf7xj3/o6aef1vPPP685c+b47FNUVKTXX39dK1as0KpVq/Txxx/r7rvv9n5/yZIlmjZtmh577DHt3r1bs2fP1tSpU/Xyyy8HHA+AOjCBKJSbm2sOGTLENE3TNAzDXL16tel2u80JEyZ4v5+enm5WVlZ6j3nllVfMDh06mIZheMcqKyvNxMRE89133zVN0zRbtGhhPv74497vV1dXm61atfJeyzRN86qrrjLvvfde0zRNs7Cw0JRkrl69+oxxvv/++6Yk8/vvv/eOVVRUmA0bNjQ3b97ss++oUaPMm2++2TRN05wyZYrZuXNnn+9PmjSp1rl+TpK5bNmys37/iSeeMHv27On9PH36dDM2NtY8ePCgd+xvf/ubGRMTYx4+fNg0TdM8//zzzaVLl/qcZ9asWWbv3r1N0zTNvXv3mpLMjz/++KzXBVB3zJEjaq1cuVJJSUmqrq6WYRi65ZZb9Mgjj3i/37VrV5958U8++URFRUVKTk72OU9FRYWKi4t1/PhxHT582Ocd7HFxcbr44otrtddP27lzp2JjY3XVVVf5HXdRUZFOnjyp3/72tz7jVVVVuuiiiyRJu3fvrvUu+N69e/t9jdNee+01zZs3T8XFxSorK9OpU6eUkpLis0/r1q3VsmVLn+sYhqHCwkIlJyeruLhYo0aN0ujRo737nDp1SqmpqQHHAyBwJHJErX79+mnBggWKj49XZmam4uJ8f90bNWrk87msrEw9e/bUkiVLap2rWbNmdYohMTEx4GPKysokSX/96199EqhUM+8fLFu2bNHw4cM1Y8YMDRw4UKmpqXr11Vf15JNPBhzr888/X+sPi9jY2KDFCuDsSOSIWo0aNVJOTo7f+//qV7/Sa6+9pubNm9eqSk9r0aKFtm3bpj59+kiqqTx37NihX/3qV2fcv2vXrjIMQ+vXr1f//v1rff90R8Dj8XjHOnfuLLfbrf3795+1ku/UqZN34d5pW7duPfcP+b9s3rxZ2dnZeuihh7xjX375Za399u/fr0OHDikzM9N7nZiYGHXo0EHp6enKzMzUF198oeHDhwd0fQDBwWI34EfDhw9X06ZNNWTIEG3cuFF79+7VunXrdM899+jgwYOSpHvvvVd/+MMftHz5cu3Zs0d33333L94D3qZNG+Xm5uq2227T8uXLved8/fXXJUnZ2dlyuVxauXKlvvnmG5WVlSk5OVkTJkzQuHHj9PLLL6u4uFgfffSRnnnmGe8CsjvvvFOff/65HnjgARUWFmrp0qUqKCgI6Oe94IILtH//fr366qsqLi7WvHnzzrhwLyEhQbm5ufrkk0+0ceNG3XPPPbrxxhuVkZEhSZoxY4by8/M1b948/fOf/9Tf//53LVq0SE899VRA8QCoGxI58KOGDRtqw4YNat26tYYNG6ZOnTpp1KhRqqio8Fbo999/v/7t3/5Nubm56t27t5KTk3X99df/4nkXLFig3/3ud7r77rvVsWNHjR49WuXl5ZKkli1basaMGZo8ebLS09OVl5cnSZo1a5amTp2q/Px8derUSYMGDdJf//pXtW3bVlLNvPWbb76p5cuXq3v37lq4cKFmz54d0M973XXXady4ccrLy1OPHj20efNmTZ06tdZ+OTk5GjZsmK655hoNGDBA3bp187m97Pbbb9cLL7ygRYsWqWvXrrrqqqtUUFDgjRVAaLnMs63SAQAAEY+KHAAAGyORAwBgYyRyAABsjEQOAICNkcgBALAxEjkAADZGIgcAwMZI5AAA2BiJHAAAGyORAwBgYyRyAABs7P8DL2KYxhmtYZ8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot confusion matrix from predictions\n",
    "ConfusionMatrixDisplay.from_predictions(y_true=y_test, \n",
    "                                        y_pred=y_preds);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification report\n",
    "\n",
    "The final major metric you should consider when evaluating a classification model is a classification report.\n",
    "\n",
    "A classification report is more so a collection of metrics rather than a single one.\n",
    "\n",
    "You can create a classification report using Scikit-Learn's [sklearn.metrics.classification_report`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) method.\n",
    "\n",
    "Let's see one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.83      0.84        29\n",
      "           1       0.85      0.88      0.86        32\n",
      "\n",
      "    accuracy                           0.85        61\n",
      "   macro avg       0.85      0.85      0.85        61\n",
      "weighted avg       0.85      0.85      0.85        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_test, y_preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It returns four columns: precision, recall, f1-score and support.\n",
    "\n",
    "The number of rows will depend on how many different classes there are. But there will always be three rows labell accuracy, macro avg and weighted avg.\n",
    "\n",
    "Each term measures something slightly different:\n",
    "* **Precision** - Indicates the proportion of positive identifications (model predicted class `1`) which were actually correct. A model which produces no false positives has a precision of 1.0.\n",
    "* **Recall** - Indicates the proportion of actual positives which were correctly classified. A model which produces no false negatives has a recall of 1.0.\n",
    "* **F1 score** - A combination of precision and recall. A perfect model achieves an F1 score of 1.0.\n",
    "* **Support** - The number of samples each metric was calculated on.\n",
    "* **Accuracy** - The accuracy of the model in decimal form. Perfect accuracy is equal to 1.0, in other words, getting the prediction right 100% of the time.\n",
    "* **Macro avg** - Short for macro average, the average precision, recall and F1 score between classes. Macro avg doesn't take class imbalance into effect. So if you do have class imbalances (more examples of one class than another), you should pay attention to this.\n",
    "* **Weighted avg** - Short for weighted average, the weighted average precision, recall and F1 score between classes. Weighted means each metric is calculated with respect to how many samples there are in each class. This metric will favour the majority class (e.g. it will give a high value when one class out performs another due to having more samples).\n",
    "\n",
    "When should you use each?\n",
    "\n",
    "It can be tempting to base your classification models perfomance only on accuracy. And accuracy is a good metric to report, except when you have very imbalanced classes.\n",
    "\n",
    "For example, let's say there were 10,000 people. And 1 of them had a disease. You're asked to build a model to predict who has it.\n",
    "\n",
    "You build the model and find your model to be 99.99% accurate. Which sounds great!\n",
    "...until you realise, all its doing is predicting no one has the disease, in other words all 10,000 predictions are false.\n",
    "\n",
    "In this case, you'd want to turn to metrics such as precision, recall and F1 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0.0</th>\n",
       "      <th>1.0</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>macro avg</th>\n",
       "      <th>weighted avg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>precision</th>\n",
       "      <td>0.99990</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.9999</td>\n",
       "      <td>0.499950</td>\n",
       "      <td>0.99980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.9999</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.99990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1-score</th>\n",
       "      <td>0.99995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.9999</td>\n",
       "      <td>0.499975</td>\n",
       "      <td>0.99985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>support</th>\n",
       "      <td>9999.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9999</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  0.0  1.0  accuracy     macro avg  weighted avg\n",
       "precision     0.99990  0.0    0.9999      0.499950       0.99980\n",
       "recall        1.00000  0.0    0.9999      0.500000       0.99990\n",
       "f1-score      0.99995  0.0    0.9999      0.499975       0.99985\n",
       "support    9999.00000  1.0    0.9999  10000.000000   10000.00000"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Where precision and recall become valuable\n",
    "disease_true = np.zeros(10000)\n",
    "disease_true[0] = 1 # only one case\n",
    "\n",
    "disease_preds = np.zeros(10000) # every prediction is 0\n",
    "\n",
    "pd.DataFrame(classification_report(disease_true, \n",
    "                                   disease_preds, \n",
    "                                   output_dict=True,\n",
    "                                   zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see here, we've got an accuracy of 0.9999 (99.99%), great precision and recall on class 0.0 but nothing for class 1.0.\n",
    "\n",
    "Ask yourself, although the model achieves 99.99% accuracy, is it useful?\n",
    "\n",
    "To summarize:\n",
    "* Accuracy is a good measure to start with if all classes are balanced (e.g. same amount of samples which are labelled with 0 or 1)\n",
    "* Precision and recall become more important when classes are imbalanced.\n",
    "* If false positive predictions are worse than false negatives, aim for higher precision.\n",
    "* If false negative predictions are worse than false positives, aim for higher recall.\n",
    "\n",
    "> **Resource:** For more on precision and recall and the tradeoffs between them, I'd suggest going through the [Scikit-Learn Precision-Recall guide](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.2 Regression model evaluation metrics\n",
    "\n",
    "Similar to classification, there are [several metrics you can use to evaluate your regression models](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics).\n",
    "\n",
    "We'll check out the following.\n",
    "\n",
    "1. **R^2 (pronounced r-squared) or coefficient of determination** - Compares your models predictions to the mean of the targets. Values can range from negative infinity (a very poor model) to 1. For example, if all your model does is predict the mean of the targets, its R^2 value would be 0. And if your model perfectly predicts a range of numbers it's R^2 value would be 1. Higher is better.\n",
    "2. **Mean absolute error (MAE)** - The average of the absolute differences between predictions and actual values. It gives you an idea of how wrong your predictions were. Lower is better.\n",
    "3. **Mean squared error (MSE)** - The average squared differences between predictions and actual values. Squaring the errors removes negative errors. It also amplifies outliers (samples which have larger errors). Lower is better.\n",
    "\n",
    "Let's see them in action. First, we'll bring down our regression model code again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the RandomForestRegressor model class from the ensemble module\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "# Setup random seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split data into features (X) and labels (y)\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Institate and fit the model (on the training set)\n",
    "model = RandomForestRegressor()\n",
    "model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**R^2 Score (coefficient of determination)**\n",
    "\n",
    "Once you've got a trained regression model, the default evaluation metric in the `score()` function is R^2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8059809073051385"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the models R^2 score\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outside of the `score()` function, R^2 can be calculated using Scikit-Learn's [`r2_score()`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score) function.\n",
    "\n",
    "A model which only predicted the mean would get a score of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "\n",
    "# Fill an array with y_test mean\n",
    "y_test_mean = np.full(len(y_test), y_test.mean())\n",
    "\n",
    "r2_score(y_test, y_test_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a perfect model would get a score of 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2_score(y_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For your regression models, you'll want to maximise R^2, whilst minimising MAE and MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mean Absolute Error (MAE)**\n",
    "\n",
    "A model's mean absolute error can be calculated with Scikit-Learn's [`sklearn.metrics.mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.3270458119670544)"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean absolute error\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "y_preds = model.predict(X_test)\n",
    "mae = mean_absolute_error(y_test, y_preds)\n",
    "mae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model achieves an MAE of 0.327. \n",
    "\n",
    "This means, on average our models predictions are 0.327 units away from the actual value.\n",
    "\n",
    "Let's make it a little more visual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual values</th>\n",
       "      <th>predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20046</th>\n",
       "      <td>0.47700</td>\n",
       "      <td>0.490580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3024</th>\n",
       "      <td>0.45800</td>\n",
       "      <td>0.759890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15663</th>\n",
       "      <td>5.00001</td>\n",
       "      <td>4.935016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20484</th>\n",
       "      <td>2.18600</td>\n",
       "      <td>2.558640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9814</th>\n",
       "      <td>2.78000</td>\n",
       "      <td>2.334610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15362</th>\n",
       "      <td>2.63300</td>\n",
       "      <td>2.225000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16623</th>\n",
       "      <td>2.66800</td>\n",
       "      <td>1.972540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18086</th>\n",
       "      <td>5.00001</td>\n",
       "      <td>4.853989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2144</th>\n",
       "      <td>0.72300</td>\n",
       "      <td>0.714910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3665</th>\n",
       "      <td>1.51500</td>\n",
       "      <td>1.665680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4128 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       actual values  predictions\n",
       "20046        0.47700     0.490580\n",
       "3024         0.45800     0.759890\n",
       "15663        5.00001     4.935016\n",
       "20484        2.18600     2.558640\n",
       "9814         2.78000     2.334610\n",
       "...              ...          ...\n",
       "15362        2.63300     2.225000\n",
       "16623        2.66800     1.972540\n",
       "18086        5.00001     4.853989\n",
       "2144         0.72300     0.714910\n",
       "3665         1.51500     1.665680\n",
       "\n",
       "[4128 rows x 2 columns]"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(data={\"actual values\": y_test, \n",
    "                   \"predictions\": y_preds})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the predictions are slightly different to the actual values.\n",
    "\n",
    "Depending what problem you're working on, having a difference like we do now, might be okay. On the flip side, it may also not be okay, meaning the predictions would have to be closer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq4AAAGdCAYAAADT+fGYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9e5xcRZk/jr9Pd0hIMpmB7plAnJ6AAt7W62f1o+JmkwCursvHkMwEkohcFNjfmshMgPGrOBAaL7BInERXVLygOHZgJh0EkRVo7YQxRpTdiBEUEIFcDMkkIVfCJKf7+f1x+vScS1WdqnPp6UnO+/UqJT3dp+rUqVP1rqee5/1oRESIESNGjBgxYsSIEaPOkRjtBsSIESNGjBgxYsSIIYOYuMaIESNGjBgxYsQYE4iJa4wYMWLEiBEjRowxgZi4xogRI0aMGDFixBgTiIlrjBgxYsSIESNGjDGBmLjGiBEjRowYMWLEGBOIiWuMGDFixIgRI0aMMYGYuMaIESNGjBgxYsQYExhX6wrL5TL+/ve/Y8qUKdA0rdbVx4gRI0aMGDF8gIhw4MABvO51r0MiEdu9YowOak5c//73v6Otra3W1caIESNGjBgxQsCWLVuQyWRGuxkxjlPUnLhOmTIFgDHwGxsba119jBgxYsSIEcMH9u/fj7a2tuo6HiPGaKDmxNV0D2hsbIyJa4wYMWLEiDHGELv5xRhNxE4qMWLEiBEjRowYMcYEYuIaI0aMGDFixIgRY0wgJq4xYsSIESNGjBgxxgRi4hojRowYMWLEiBFjTCAmrjFixIgRI0aMGDHGBGLiGiNGjBgxYsSIEWNMICauMWLEiBEjRowYMcYEYuIaI0aMGDFixIgRY0yg5gkIokCpBAwOAtu3A1OnGp/t3AlMmwacfTbwm98Yf5s2DZgxA0gm7b/btg0YGgJaWoDW1pHvHDlcwgPdgzj43HY0nDUN/3brDDz+RLJaT+lICVtyg2g4sB106jRsmnI2Jm78DU5+bTsmnD4N0z9uXGjdOmDzZiCTAVIpYO9eo/7GRuDJJ42/nX46cPHFwAknjLS92o4jwB13AM8/D5xxBvDvV5TwzPcG8erz2zHpjGl40xUz8J3vJUf+/u/GPa9da9Qza5ZRAOOztWuBchk46SRgzx7gxReBXbuAyZONOj/9aeDxx9l9Bhj3vekOo/4Jp0/D3rfPwM7dSaNPSrDdb3MzcOqpQEuqhFceGMThv1X68stn45m7foODz23H9tJU7N0HHNmyExNOn4bXXzIDs841KjSfq9mOUgn4r/8CHnsMOHywhBkYxFkN23HmjGl412eMhpr3CBi/OXK4hOLNgyj/fTsOTZmGkz82A1OnJXHqqfbnzRoPp55q1FO8eRAnvrId6bdNw3/kZmBig6VDQhi31n4ulYz2/+pXRj9Onw6cc47xd+tYPvts4/fW53z22cB3vgM89xygacD73ge0tXlfe9Ys+zMGRsbdc88BRMZ40TRj3G7aBBw6ZNT31jeV8IdvDOLEvduRefdUnHEm8NpLO/F3moY/Ns2ANi5ZHYPWd8/ZjpkzjbH1vyuNa7X+4zSkL5iBXa+MjK3BQfuYdrZZta9Z9/v888DrXw+89a3A+vWArhvvyY4dxr1/4hNGn7H683WvM95v87uLFtnfaet8NDVdwkmbjHfiZW0ajrxvBl7XlsSMGUZ7RHPa4NoSnv/hIHZt2o5Xm6YhNWcGlnQmMX68ZH9U+vl392/HdkzDzjfNwIFXk9i8GZg0CXjve4FzzzX6uFSyzz+f/rRx76L+PHIE+PrXgZ/+1Bgzc+YAV18NjB8v9yy8npvXOGb9zpw7BgeNue5d7wJOOQXYvRs4+WTg978Hjh41nh0R0NRkf37ptDFf/+Y3wJQp7Gc7uLaEF+4exPCL23HC9Gk4+O4ZOOV1yeo8Yz7XLVuMObZUMtr7vvcZY8c5f6ZSxtjbunXkHUkm7WuEaKyI+jZGjDELUsCyZcsIgK286U1vUrkE7du3jwDQvn37lH7HQz5PlMkQGVONuyST9n9nMsZvRL/LZIhueW+eNsP+hc3I0FzkCSCaC/ffjyLJ/b5ZEtBpJoq0ADmaiSIloHPbnskQzZljvwdWvc56WHU0NBA1NPD7SVTMPiMi2tCdp21Jcf3OYrS51dFXCW6Fm5GhRSfmqSVlv4/GyTppGr8ftmjG79x1i9ubyRB1dxNNb3X3G+/3t7w3H3jcOuub3qpTdzdROs3umkTC/W/Z8WTeI+/a6fTIMyYyvut8d/jPlv8C7kAzLUcXzUSRWlJ69d1jtUPmWYnarDJHWMe0yv2apaFB3J+8YtYhutd0Wnzdds392x1opvnop+5u7/7Y0J2nLZqz7lbqQdY1jk48kWicZh9j4zTdNZdY+7O7m6rvqbVomjGfeT0Lr+fmNY67u92/a2hgt0mleL1rrOdiHb9ez9VP8bomr2/9Iuz1O0YMP4DKl5ctW0b/8A//QNu3b6+WoaEhpQrDHPj5vPpkZP0+byKaizyVoFHJ8eNSpdyKbubfy67va1SCJiS7u5CiHmSFBNYsXu1aji7qQVZp8ZftM00j+tEcXv32+3S32d03zn+7rwcaQpp5H/x+cPd3CRrjucDVXtazGUKaypXvs+rxS17zeaJ5iiSNNx78XoM39vN5Y+GXrZ/1HHhFtMHqQVZ5bJlFtDDz5ghzTKvcb1hFdvyq9nkZxtxUJa+6TlQsEuVyxv/rOm3oZv/e+Y5Y3zWZMWb255w56v1hfRaqc7uKISDoM/MyZPh9plEWVt8GQUxcY9QDoPLlZcuW0Tvf+c5AFYY18HVdbGn1OxG1o582I8NdjMuVyUhEvJwT10too3YMCBf5IaSFk1sCurBdzjaGPXl61W/ep3XhSECvkj/VCsvc+zD6yqsd4zAsrLsEVNvLW3RYbXDW8+oBXXncXpEOvsgFJT+8Rfh1rxNbHk2isBB9tAMt0qTV2TZWG7z6WmRN1hmPwWuO0DSi1lY1S6tXv3gRKD/vkexvzfE6XxugI/e6zZXl1lbarfHfHda7Jtq4hUnGNI2orc14ZrJze5CNm0rxete81gyv8Rt1sfZtUMTENUY9ACpfXrZsGU2aNImmTZtGr3/962nRokX00ksvCX/z2muv0b59+6ply5YtoQz8YjGqiSia2WMPmoTXNgkxb9KdiQA3HMLkKVv/TBSr/5yNQuj9KPt8fo6PSH1vNgrSGwJWWXFBUW3cFvwTF7MEIT9hE14/43An0r7eNevYcpYi4zEEmSNUigqB8vMeqf72FTRSGW5zpeoGUmbjFjYZKxblnlutLJwy79rLaPH9TGtZWO+IKmLiGqMeoKQq8L73vQ8//OEP8Ytf/ALf+ta38MILL2DGjBk4cOAA9ze33HILmpqaqqWtrc2XL64T27eP/HcCJczEWizAKszEWiRQ4v4ugRJWohMAuSQVEqBQ2sbCydgnlHDQAACEFehitn8atrs+U0EChOnYgs/gG1L95Ld+6/dmYa1qMz0hO2A/il9IfW82foU2bPUtr/Hq82rPpbR2UFif+ZxmYJB7jRnwdw1z7GuCsc8bf3OxBqvRgVZs5bZLBgkQWrCb2QYviMbgdsafWJ+FDbNfMo5+acU2rEYH5mKN7XM/75Hqb0/CfoAxl2lSv7Z/n/cbmXHqB9u3ez83mTmcN45VIfOunYIhqWvJPj+V9UwFtXgfYsSoBZTWjn/913/F/Pnz8Y53vAMf/vCH8dBDD2Hv3r3o7+/n/ubzn/889u3bVy1btmwJ3GjAiJgEjIXjRZyOtZiNVViEtZiNF3G6a8Ew4T0RjR4SAHcx2I5podSxAkul+slv/WG1s1Y4DcHG46Qz1O43CHEJeo3r8WW0YasyGRERBb9QJVEAsANTuQv6NMZjYH0mgiphSKCEO3EVNJDrfngEKsh7pPJu+elfPwi6oXZixw7g6afF3/G7cfODMO9P5vmprmdOiMaw6vsQI0bdIqjJ9j3veQ997nOfk/5+mD6ufnwFFyA3emc1kmUBcoIjq4ChsZL9pFo/6+gwCleBkfrYn6seh96Nj/vuO18+roWi1PVFx4p+jpvNIDmZ3y3Scr7qi+5ZG+4FvON4Lx9XXpCP1cfVj79kD7LKz8HPe2T97Q40j+qzEN1b0CLrayw7h7PmUdUiO/YNf2/1Z2otft0fTP/q5eiiHQ63hc3I0DzkYx/XGMcUEOTHBw4coJNPPplWrlwp/ZvQBr6u06G0up/frFFehGUKbzEYmdjUyWsYvmq8+nkTq9/gLC/fuhEfyXBI/FEkPQNeWPfrS1WgOm79ERd7YJTcNVQC+wigL3zQPv7C3Oyp+rXKBAlt6OY/BzM63UlerdHWftQyEtBpF1JS9+AkUKrvkbW0o9/j/QjvOaj4uAZRFVAtQfyEVYvsRmMk+Fb9mdrrketv61gS+Z3LvCMqiIlrjHoAVL587bXX0tq1a+mFF16g9evX03nnnUfNzc20c+dO6WuENvAlIy/OSdgnr9MyBnFgBS+YL/rWRIb2neAvGj5IcU5ObW1yOq5hFJlJPgGdepB1LdgvoY07Mc9FXrgAssrOqgwVfxEIsx/ExMgty/US2oLpuObzVOYscmVodEmDuy/bMeCypshGfMsu9GWADqXbKOlYHMOyuJpqHK81pKU3HTuREqpIlOEdMs3SA21rq0gEVcyyIiI4hBTNRsFGGv51onyfsN4t1vg136OGBrE2523oZra3KjuXTgvNzK81pGmrJlZ0sG8YvMlYtT9JXcdVRdWhpUWOTG5J2EleEB1X2Y0GS8fVOjdaNVedKhSyp1PuUxTvIEeZd0QWMXGNUQ/QiIhk3QoWLFiAxx57DLt370ZLSwv+6Z/+CV/+8pdxxhlnSLsm7N+/H01NTdi3bx8aGxsVHRssWLXKSF3iAf3uHH7dttCeeeX+NUBHBwgwpqIKTE+18sBqJBMAdXQA5PZfiwJlaNAAbFq2Gk+9aZ5U5qwTH7kfb/rvFQDsPm0EdR+3hcjhHizk/n0u1mAlOtFmCULZjRRWoBNfwRdwfU+ymmXnsWIJE58YxBmTtkN73TRMO2EX/u+qpThxFzuwx3wCv3hzF7b94xwjc9b++5FY2glt68hvKNMGffkKfGPrPFfmrHdN24E3fXup4l072pFMQiuN+IQdSrXh+c+swOZ3zwk/c9aaNaBO9/1pK1egNGeeLSvQlc99FjMe/6rrmZr9Zv28nGnDzz+0Ar+YOK+aOesfn12Ft37J+10hAFel8/je7nm2zxMo4UWcjlZsYwYwEoD9E1rwg7f14nX/51SccSYw+VcP4rRf/wSTDo0Erpj3B5jvFqCBbNdxjmPn/XFRLI6kh2OAm61p7Vpg9myZGrDrxAxy71uJt3xhHs7duQqJi7379MiUNAZX78DO3Uns2AEstQzRBIzxOw1G9qpBzEAZSRQKxq2IMmf95Uur8YbbP42Gw+z+RUdH5UPL89IqPbl6NUrnz6lmzjp593P4yObvIvXqyFjcO6UNm69ZgX94G5Do6oS2zT5Oy70rMNg8z1/mLEvmvb/TNFz0TeO+RejpMTJ5bdtmZBg0g+KMfhy5x3JltPy6czXKF8wLLXPW+AfX4L19nbY57HBLG5779IrqfC2bOevPX16D16/oxORX7HNpGnuEfQAAf/xcDkPnLcTQyyVcsPR0TBji+6y74PGOyCC09TtGjCCoNVOutcWVqwEiNMMIvhNVaWnxpxLNSvmTSBBddJEhPN7bq7yTd5b2hPgo9cp0fmQzz0t3c889xj3yKmGJDTIE1LnwcmiULb29cvWFAZn76++XGzt9ffxrSL4rf7s8ax9GzAQBgjN3lfsL+93K5fw9g5yCG4T1XmXnn2zW1h1ePrdKhrFK/5b6crSxt0ir+nQqFIgKBaLHuvJ0uMVjfpN9VirvodfXGc9dRnvVfLzWbhdZrcOQfvK8seFhpX6p3r/GTooiNZ7MG/Oj9eb3HbEgtrjGqAeg1hWG6ePqSVTSaW+y4zXxmN/p6nITr0xGfCynUEp396n3gUxaII9+Knv4uMr4Xh1KV1ZbUXtUJ2Y/4Dk01nhyDw26TtQsGYxj9ps5Xvv6DBLe12cwGa93JZOhVX0jY4CXSczpNiEkQzL3J7mximzcqBIAk10OD/uaf2R8blUg4v8J6NTRXKTHu2q0EeO0p7nZ2H/x5gcZP1Dr8LZ2OyvxQyJBdG9OjWyHcqNe+VU9MisIXaqcuxqVDVcYc2sFMXGNUQ9ArSsMdeDn894va5iJmllENwyyBFBHc1GtqTJpgdoshFIwWYoWDGn/RpMcBegDAoITR54lPSsXAR6NqcYnVEhVLidmMaZznYAxmdV5Jej422XZ8AiBnwWYN85F4G1Sdd1fAvli0fvd57zQMoc9MvfyeFeOZkGc5jTslJ8iiNK0JqDTKw3iICLeBrqlxW2sF023TP93L1IZxo16dbbk++wyErCuq7rh8jLiSCImrjHqAah1haEOfK9FJ8xcdyLwVqL+foPQpfjRx+aEnYSutsCouEoEIK7SEeU9PeqLP6+9QcEiKaGf09YAKqQum5XbPDnfFwtj0nWi6a3eFvZyhtNPikfKRKS2APs1U4qsY36Jq7nBYl27pcU4oREcJ4t4tLALfRy112Joe+2jgygBdHW56+vvZwd1cQOWwmLwKgYDJyTfZ5daBWtXo+oaFRPXGMcQUOsKQx34Qf1cw4SXP5+mUdkxyTiPyJQWGFlS09cnPp7SNNqadEeSm0VaPiwoca3F6hr2Oa1PyPI7Wc3XcnOznLVb04zvFQrcytdl5ep0vVN+jk7NzpDZUAwM+DNTelnHZC3xovsXuRM52ZWgTzy7MMBRe9TToNdUHER7VTadr6fsm8wco+vG+9HTY5RCwf79IGuOrBIOCjQLEi4eXqbnCAZATFxj1ANQ6wpDHfiy5K0e/Bbz7mAJnoyU1PwiO4FK+hDOQpHJ55Iw5MM8iUWhID+BjiZxDHxOG371PC5TLIgXYtMn7m+X3iTf914DzM875ffo1Pl7r3GhatGVsY4JTkO4hWW9kiURnD7x7MJ+8b3I6DFHOQ16DRu/FlcV46W0W5MoYJdlfU+nR55XkDXHY5NmPsPTMrr8dJTPy4/hODgrxjEC1LrCY9biKoFVfe5AAt/zi6ylqq9Pqo8e78rx+ZwMsYjachYm/BxphwBVfpfLibUaywDdim5av0TRT1Q0wFTfqSBHp47OKTuuU84EHBd+ralexaIUINUHHn0i04XzW+Sei0gdZDQtrqoZw/y4i0q7NbHGv2y8hIrBgDXmOXNpuaLhvCmbV5+OZI0GscU1xjEC1LrC0H1cFf0WI+Es1ouaejSMCqxzHisalju/8BotQygViIiqjA1TPiwKy9kxAD/8zhos5Qw2eRkt1I5+Aog29hblSZPXAqb6TgXdPFoCjS5MF2g2CtV3YnqrguXJCRkiYpZUKpivoB9pIkufyPw8yFF7rXxcvQQwzE2YM/lLmeHqkMkY+wPeFMGqz7fFVdeN3L9ev8tk5NQkrN9nDeCwT31q6L8fE9cY9QDUusLQB76C36JfNzzP+kWMxFKBOb/M4+RFZ+aU9mq01yQY5qQmQzijOorXddILRXqqJ0eFniIVC/qY4rt++J310VnTvXailxahj2ahSKdldNKHJS1+CtZPaV/gIEenHoFGvj1IVC2gZmCbDBlhNcavMkKlT1g/DyOzkvWx1eJAQ0Zy+Mq027JObW2kD+SrU0s2KzdPO+vzsupyx7/KxkNGTUL0vpgIe/NeI//9mLjGqAeg1hVGMvAlyFJQNzxuvYqT14ZusZi/Lae0oNFlzThWyuUMX0i9UORPgrUOSlKZlCXJ8KG0m+Bckc7X1MMgCPzyO+ujY1leD6UrK7rsWJTtMNE7ZX1msjqsrGAujSXEHiBg0YQKEbFKxonIrmjzpeLfzegTZ3NZz3kzWmm4ka8ZLfJxdXo2RInubonhJ3jnVedpZ32i9KxlcMa/ysZDpCbBu+laqZXUwH8/Jq4x6gGodYWRDXzBZBiWG56rPlmrjllB5ZiJJzJdBkayIHnoojoXKk/L8SgHJUm3yXkj+XzlKJFNcOYhIHmtkdtCkBP1fJ7oirSEzI9oMZV91l5uL6w6RMnmWS+Xx7vDImE8l0EmVIiItU8kXX5cUCWuHB9Xc3PC29iWrb93vQugHmSZrke1jk0dGHCLK8gMP7/ztLM+XkYt7lyhanE12zus03OLe/2/2FEg4vksJq4x6gGodYWjMfAjieHy49cWVpYgSzGPBqWMp/XkWypjWtF1KmfEmqJmFK6vW4nEd4SNQB4blX6QyqpjPmNr5qzKs/ajEWrTO1UNduINSsl3x3nsLf1oZN/Nyy4L5x3wmzbW0fVJGTmndNr1jHYyMppZXS5GIzbVz1QTZJ42VazMAHtWDAH3HVPxca382HxVAgWEhY0azO8xcY1RD0CtKxyNga96TCv1/vvxa1uyRP03HsUajBHaqVTUE6CsaUXSkjUTRfXFWfZMMsS+8O2xEcLOy69GaPUzGckdp+WVZ2aTfHecgUbSni1euwRuJ/iEqmsCpz5pDd1CgahYpFJfjm6bkhW6Hl2Z9hGlPkoIqm4o8xgS0I1gRlagq9ePK8/N+qoEluAKC4wX/HBLhvSBcDfhMXGNUQ9ArSusd4urtBGuziyuoc2RtbBCyvadZFKDBci5FjMh35Qkzvq9Ay7t3aB94ctjI+CKHlQjVHmMe5F8nxZXy6PxJmMKATQ2f3FBs7mQIcqplFvM3gkfO+xDafGJxKF0iIGXESPo/syr+zxTwebznjquzqnDd0BYmPDwF7fFTQRETFxj1ANQ6wprMfCdc7CXgok5twwMKAQGqFh1HD6u0tI7giIKxvB9KuUV4JPN1vZoVZK4Oi2u+X6dOprtx4Q2vim5QpbhzhteRvCANmWOEGBFl+HoshqhUkVm8Hm8OzJi+lKbM8kAGmV/cV5dQQMgVZ9zWD5QNXSZESGoAIqoO6RTwZo+B5zMWaw6eAFhkQW/MjqN50Zkju18fzjEOSauMeoBqHWFUQ983hzc3S1eV/r7fQQGyFh1nJOXrCXIY6EVpXj0ZXGVDTar5dFqoVDxcRUTHKuP64ZuvtRY9TFIEmd+EF0NI4WJAq3oYWqEShXZwcd5D2TTl0pvznRd+qRDyV+cd0+MJArS1lzV5xxG5sBI5Fb8Iwj/53VfKKlgK+B1OdOaW4vgV8m5dH5LMZTpKiauMeoBqHWFUQ58rzm4u5t/TOvbeJFn6BJ6TV6yUiqcwksVa5t/VSWpVNwYgi5oKgt0VVWATXCskcL6gFhqrKqTW5B82MqDIUL4XNFluI20n57XmFAl84z3YP9J7LEdqOt9+NT6PuG1vHfrskYSBWs1Uuofss85jOQPocutBEcQARRW94XphyrqcmtA2MbeYm36TWFshzFdxcQ1Rj0Ata4wqoEvOwcPD7P5XBCdzemtIxPWbBTowuYCPd7lQRit0d8tLWIS19pKdPvtREuW0B8u66UTMCxe11SO/vyQ6DAWNMkF2pSCYsnbXGnVcdV1OtzirUCQgE7FgtdRtWQ/hBQpzN1jOP/gI12ubMCK0E9Pdkz42cw47lEf1sNPAhTAp9bvYh/IkCnL3KI8Ww+jEwIgiMuts/vCjPwP2uWhQ2FshzFdxcQ1Rj0Ata4wqoEfdA5m/Z4lqWLzpQzrlE1E4gBXwMChtCG+z1zXVBrl5dMa9YLmsUBbm8d6Fv39lmupTuCcPmdpxka5oPP2GBu6OX/o71da0WUW2lSK76dXBt9lwtauEI9Ew3AXVemEEowUuovQF4oGaiiGTFnmFqSzwnA1GE0I+sj6J+mUyJLvc+jjMwiqG3axO5Vz7fKLmLjGqAeg1hVGNfCDzsHOtY3ls7QtOSIvEvopG4vBsCJcKxdnRkKrNGp42K0SrlrCWNA4i49y//o5MmNKyLRRO/qFFsgSNDrcEtyswts3zKvmdGfctI+V0WuhNeVZWWN+J9KVIDUO8w0rYI/R5lDyZZjjq6tr5KZtz9JNzINqoKpuogMH9Ofzbh1Smc1EHVtcPSFzqiR5qqWkviCofrTyuYy4SPHdqcKyAsfENUY9ALWusF4trkQjC/w8TgSqNaJcqT7Zlcn6PY/MWUxmLNuobJaouVnuu6O0oCk/T79BCpyj6nmC1JElaIH1EXnEPMxAEitEC+3w8IgEK8uyPVqBJ6EQOo8sXyyLskgDVaZNKpvoUAL6/V6kKqdVL+fekpA5VWIFysHtBhQkA5+HAEFNsf46frawMK3AMXGNUQ84ZohrWL5H+X6dtiW9icOqPnvQBWvBB4ge6/K5qPhh4n6SIvgtES9oyhb0qiyM+MhMRhbGuoFhLQZh6CLyHm+UguY80hVIuL1e4eUGc/XV9Fpji5IGqiw/VNk/BnY1CuCvlM+LN2hl1PrcWwIyRzHpNLNPWNZ1M9A1pNjC8FXEJHZKZjtYa1DY+8uYuMaoB6DWFdZCVSCQ75HkqjOwuFj9J8sitRkZuhXdbCIVlf+Zn6QIiqU68Xd3B3xaoTwGO3+rCnGzraROwilaE1iLwfyWYmh6iLzHOxopJMe6q6MLMuRG1k2mMsBU+KHMJjqTCckP1udFrD9lzV9m8GPd7U8CznEif2bXfCJATVTEJJix1/7MFgcQAmLiGqMegFpXOBo6rkq7TgVfyWSSL2xt7u6l8suz4Ie5ea2YIZWy2akRrmq+Legcv1Xn0b6Ki1wURsbRsLiqtiXCKqNFmBu4XI70YXdCC/PPCeg0C0VanMoZMmuVQSLrVxyozwM8OOdPeSdGvp95VC9PSKdKLAUJwDjy92pqaPENXjtnD2bs1Q7AMD6HOW/FxDVGPQC1rnA0MmdFJZ3j6Y8YZGXyy9y8FArCLCEwGZm5W9mC7jEA6kFznfd4vaSpytConAm2aXB2j2xmubqzvjGg60RP9YToMpPNutL+msFbXilERZvoUKzcAS4SppXd6ev55DKGtnUIZ+i6rqAQ4FGsmr3O4iXeEcpGT7RzlmTGxQI/s5xjCIeGmLjGqAeg1hXW/cD3lM4ZkRcJRbhdJquNKnPjrZiyZh7JUurzWNUkCCRz7u4f+R1LxD2I31boahABwHu8I6oCbJeHK9LqgSTWOv1klqs3V0cWzHsLLaFCxVeSHbzFDvZxdlgQv2JP8hOixdVX/ZU+t4qfSKdWVYTVdSew5jD4FldesfLuwKTfa+csOU+vuEDuHsK0utb9+h3juABqXWGUwVmhnUxxGIUzHWUoqTJlVgY/vg+sDgnZlWBjr6DtHmfxIikopxWr3JqRT5vpgXo7Fuc93g3deTqUdvsdmoEkfjiA13opyixXEwR4iZ2av2KrtfjfVSkynhwdAroBUUjBpAEuEkb9+bz9N1EqYljbydMclillgF5CxuXbynOVcL4j+XzAOURm55xKSVUgshpHNZ/FxDVGPQC1rjCKgR84upO1YDIu6ky1GsiyozKJh8nKuaTcKMvRRbNR8NQxfQmGsoKwDtY9a4aUFGvu5vsLhxfhHNoRaYjPhLfHsGZkcy6mqhxA1tLMyywXOQK8xNZ7MwnIcnQx9Wc9kylU3vO/XRbC6YQHWwglmDTARcyfJh2kLQnds35dd0vHRuGfzRu3TDcNyYC726ZkPa9l1fJlvSO+SX+I/tcqVuOwAitj4hqjHoBaVxj2wA/sr+jha7Sxl78L97LsjBwnOhfPUT5/ZdzzloSdlPOsGlars1+LwuGWNkFfsn9nyhPpw3ogYhW5f1pICNsyXHNLswqxD/gSm/fGIiBHkZS78UrpRC8loNP6JSGcpixZ4nnvoQjZB7jIhu48bUvaf7stmfGUfGONpygUMUTj1mol3dhbNBINSNRf6stRT49znnPPN9bTNec74nu/ILtzTqWk3NVkh2JscY1xLAG1rjDMgR/YX1EhcpO3u676I3IsmPfjfNoBuyXgYDrY+as+bBDq9UuMCVsfVjeL5fvdkdLNzUaCITP3AU/HVJiJRSG4zfqRrLWmo9n+O1W+GPiItEaRXWFLVNVU8kqF2Hu8xDLBaLmcWN2jBNCD+IhUB5jHr2EFAckM0lCM934uUpWPY/S5x1hmjacoLK5K41Zhd1Ysym2WWQTRfEd87RdURX493NW8Stg++zFxjVEPQK0rDHPgR+5rVHnjvXbXrLzyTkvPDjTTcnTRTBSpWPA/i/i1kJi3XCyOZL9k3bKmGRG1ZnyA0/fL8xhRQU7M+pGstcb5Oz98kfc8zVLVPuSF39dglRizFldVYi/ZsHVZfsOKBW8C8jLkjpFnoWg8wmHxDkfo4yp776MJj/nPKw0q67F5Bk75eD+Uxm3lnmSSkOg6UUez3MWdm2ynAqHSfkFl5yzhrlbrYRcT1xj1ANS6wjAHfiArkuJKztpdt7QYJLBYNBY6vVCkOxu6hKkFpUS9ObPhhm7xsZaIvLLazytJwelqWBaFOU32xUDWWsPy6/LDF0X9kcmwNyPSaXJDkgkLU6Iq7OsJKxGtpM5KJF/ihchxx92RR4pS1yg3i/PVv4Q2SkIfqUdGWk420LGWchUyCLhhYPm4AoLAKZ8sSnXc6gNiFyfradHjXeqb5VAi9FX8DCprwVM9bHc1UYkisDImrjHqAah1hXVjcfXBeq0WS2ccgCnjZOT+Zl+LZcVwcdQB9jGrfk+/MBVtCRptTbZV3Qas1+WlllQt2Wxwi0K50k7nBDwOw7QD4hScXn5dqnxxYIB9nblVOSqfHSXh2yiDUIJ3IryeC35eSMnfVC2hji7N54kWpyTf5a4u4fHrlSyZMcYO52C6jTZl8+x3VeXeRxOS89/iVI47jJ2qAtb3xxU4FYBFqYzbYpGfCcxqqSwWyZdbU2iaqIp+BjIEPpMx3LyiDKyMiWuMegBqXWEUPq6RRnc6FhrRSegsRR8v59zFJUwMPzRe2dhbVLKuqpSWFuO03BOclabM8c9iLnSWIuvXpeKfyTMOhpJUwiwyDrg+tG6DWFLCvp4Nfo5AJI93zQ1LT49d+EPTFH0rRUSUt9DrOq3LGtmxrFav5maipVcb/ualTy9Rv/fRhAJp43Ftp4arba5IGX0WFosSjlvLO1ToMZ6Pl7xVLkfGZJZMcufWMgyXr3EYJiD8LFSid58ndBPpxlMCMXGNUQ9ArSuMSlVA+WX2wXq9TkIXKkTVOgmwF2GSJa4/+kguFOsqr7S0KMj0ODpra9Ltn8UVLLcUWb8uFWMWb90ORbxedhBKBjGFqlEcwfWq8HsEks9XNzXW74k2LK2tI6RJ1rfSVKRY1Vchm31yHcDbrFqLrL9k3VhcdZ0Oprwl7xLQmVzbq0+qfuLhNtk9bhnvEEvKivkYFC2uyvKKfX1Evb3G/yu8aKJpIdKNpwRi4hqjHoBaV1grHVepl1mR9XrNc7KkRy8UXe0NizA5fUedxcsSIVOkd/eWlWZjr0hOjF1RCaCX0VK1eIjao+o+yFPOCSWphEzjaqROUFMEOALZlPU+3hUVnm+lKT23odutHSxrEJc5vUhWxrJwg1lPPq5k9LmX5B3g5tqB1VzCAucdEm14bG1TcJeQfh29nOc9LiQzLfA2npFtSC2IiWuMegBqXWHdZc5SYL28ec4kgwvRRzvQwj32NGdNVo5pWcIk8gHdkhD7gMoKbXsVPwsTS8kgSECWs6jyvN5e9nWkNxCSYufV4gxFrouVPwL4PAIxEy7MCrCpYo3vg+k22tCd971HUNGL/090i9UGurvl+7EGLETXia5I8zcMvGFYc01gXuM9VBGcPvGu5y15I3qBcyPOZzQw4G2a93gH/E4LNZCWJqKYuMaoD6DWFdblwJdcJFjzHGuxZC5ellkziAYiK6GBaWH4/vl8AiojtC1SEwiyMOk6OyDfrwSWs7B8z7we6d13s68lLeljppha4sO3sS5W/gDw6lyfRyBeMmUyxXmiUHhYD7RHkHXblfKNlt2MhMRCZKa1fN6wFjs3DCJSL9MnCehU6PGo3E+DTfgIrHINwSBBEqxnJDOBCq4ZwNOmZoc3dbl+xzjugFpXOJYHvnOeE4meu4irZdb0o4FYhjHhbbh2wKXjujVpWJV4E5+3/6yRzWrgHl2JNDDjTBiLT1B/UhmLq3Uy91r383mxspWSpI+f1UaWDfX1hTZ2Q4MsqfJpMeSdtKq6uJj8oFDwRwbMW+BZ5v2OZc/NSEgsRIX7qu4zvIY8M9jSi3irknXJd+ipnpx4CPo5IZBxepYdB4zAMq+fMmIbPd+DsAz2Y3n9jnHsALWucNQHPmtBrXxW6jN8MVf16UILhaaN+LR5+Wc+frXbMZ+30fdKs6oPGJMoL3MW77oqi6oXqROuwZzF57EutiXYO2WufGpDazYb0brf3S235syFO9EDczX3Y7WRJbvSkXA1Qo1MO+Yrak3LqeLiYm2OX61nVWWOUNKdhsRC/DwmlX2GaMhzgy39EEHRb8I8tVBh7rJOzzLjIEhgmaMbvDZ2YR3ejPr6HSMG0XFGXFkTVDrt0nTZjAy1o586mouGSLVjJs/n5aOI57cUhQSYRV55Pmeyc7DzuqqL6vCw2IWTuX4KFp+yQMrKi6zL+t+aj8hrTZE9zdM0Q5dXajVXtdr09ys2RIEQRuUbKUGqyhnDfztM9YMr0mIXl0sa3EFXVs7hh9/4MaiFYnENgYzVygLHGvKe7hKsyv02OJAWIqfjZN4bFadnUeGIa0sHllVgpjv22tiFpcIWE9cY9QDUusIoB75w7hGsRs5jfeZRv+PYqtQn75+pqoPI2z3LTj5Ofu5nUe3uFq8lTF8xzg/KGjvpgFlkBMNl1rWw1hRfhs58nsqOPihnGFYbVVaksgBHGaHhw6cwcNW6XEIPU+aK9d6r8BtdN1wLUin1MTOiKhCASAVKBaj0mEKxwIUxzwRq8GgIm8o+I9H7nMkEDyyrYEQZwnkNo/CUIfwiJq4x6gGodYVRDXzhmu3jeMdJXMvm5JjNGpOXpPPbTBS560y+Xy2KWmXysZH4gm6QKhFhsmQX8OJWruBoycUnix7ufSag02wUKIseyqKHZqPg+p7X+hR0TTGLH9fSfN6IirduOKa36koEX+bhczdnUR7j6/rIub1HsQbSBa46JBYmw29UXQOamuz/bmurpAkOQKT0gtz9cqPcKRTuqwTreHyqx0flQRtca2HToLtjTTPWEInvCgPLKp1fzoi1v3ciTadl9NjHNcYxBdS6wqh0XEVr9rpsMdhkw5sYEkmpFKXMwI8Btw8lz78plOM9mZDtTIb0AffRq7O42qLIGFn36XXclc16r09hWVxVrRPSnDFIAytJK5ibs/4Qz4dZEj8KjM4ZSBdo7IbIwkT8xo9rgKkpz9xA+CRSxYKXz7fhN198mJ++blQFK/xUHkaDayFgaq1LNgjAWdJpJcdrz8Ayyb7700XZ0G4/Jq4x6gGodYVhD3wvI1YSOq2YImctUi2m7BXP/24e8uxFO29mCmL/zknqQjv18jIrefijctcRRUJWhlFPriNPDz/s7cd4ZTpvjaHjTuZex8KA4VoalluctU5BlyoLnrPKumyRS45VUw37Hh+C4hVI54sohczCeHGZfm5ZWKWzIlM6zYNYmf6KTEULSzmY4vtghO36Kd2Rfiuv/IbnYmGqqdSVpjFLlFqmFArG78Ma17LzSYi5amPiGqMegFpXGPbAF80BTFmWkIuZz9r62Utoo3kVAW9XgM/wcGWi5hEAw6qyCH00E0VqSenhnnp5RF6VJSP5bUYuGcbIW8SGh6X8GFXklESntaaqQFhucUprkA+La6kiVTa9lf88VFINe3acj3dAJpDO19F0CCzMa7Oj+kiUiZ+C37HZFq95y8wGxhuskbp+ymjNKVa+oVscoLmhu46UNYiCD5qwdhcq7QjJxB4T1xj1ANS6wrAHPm/TyZVliajc2Nhr81Wt+rw5J3nFY6bDzSEF15jwEWQjNQ/6VY+XFcr0mHjNwJqeHqKODnc3W09rlU9zBexH6TTb07pk/7e5cOc6xBbwwBHtASV+ZALpfK+bIkLtwcJkOKOKEVyZ+HHabp44bMrmuYFk4zBcycLHbkwZGu0/qY2GX2WTm0hcP2V9YhQqN++ZF6DJPbUyf1wrFwErZFVBROO00pdlR3+WVQaZrstHEobk1BwT1xj1ANS6wlpYXKWy2FjIgisQy8fiXerL2efQAf8WLN8TmQwkV+qFnGxVFkOpe83wc9TsJ/OUAzx1hsZG41SP51YgteZ5sB/VUz+edYk15nYiTXOR94yLSkCnIaTF41Z0XBjA97Z0fQ9Nb+UnrvDrfmF7Ntd1u4lCMilMoSrLsVRuXYn4eWwGTNcKZwCf2W5Z949zEkVuN4TK61RlqyQrt/a/SIuUuVGuRY5TJ/ycTGTZPqYbut1xDmYyGWlIBnrFFtcYxxJQ6wqj8nG1ziXSFqgKORiCnfWw/FaVJoawRKqDrP48SK7Us+D2qbQet4sUHDb2FikLSb/igBbXfN77p6K1zI+Emmkx0wfyynJLPOsSS5KtBNCV6bxn9qfAxDWIHEOxGOrRtJOPmCcnohTKrGcqy7Gcz49FnlIpw5qv9PopvmdO8ro4pZYOWcDhw0FEUV++4u9qlAjDBb/zOmPTbd6Cc7wlK+l2lTZIDQ1ygz0ExMQ1Rj0Ata4wSlUBcy6TFdzPoocS0G2Tx8DiIg12DtDeRjGxEE4MYYW3yy4KKqYVSaaV73fneG9r42eesq4ZuZxcVqyD6baqz68fk52uE7W2endbJsPukiASaiUY2rT5fl2auDmtS7NRoF1ICfydR3RKRV0UODjLz3h1PJcwjqadfMSXoL3C7RSLxnNely3SQuToBmSZyha+fCwlGdkC5EjTjL4rFEZe4SOPyN2E6dKTTFbV7KJBRDpbynxY1fIbJvzO6473LtRbCLpzV0RMXGPUA1DrCmuh4yprcbX6cTqtMFZNzh5k2ce7vB1+WIKiMouCnyMzSabFCo6WmXBNC6FXVqxNWf8BHUTe64h1Q7Kxt2hbCcKSUJuFIlcH1EncnMNCNRUvr4sCB2f5Da6z3JzVx7inR91CyVrM/fruyr5+j3W5H5rbbcinFS+gL/n0ViP5Au+ZsFQcenvVmhjF/VQj5yWhHKcUUTuk4GdeZ+yaQzNey1iAQ1QUIIqJa4z6AGpdYS0yZ63q0+lwCz9y3/RrbceA7U/Z7Mg1nD9jHe8ebuGYlGplcQ1yZObDRKayZpiLES/owpS4UmqPgx398np3kgLR8zIJvYzFQ/aodiFytg2PSgS7aipeXhdJ6xQXCvwGqgbXWc6mw3A3ZI0t1f5RGadzkednuWINCL86aQrE01nlvEobne3kqTgsWSLfPFGzmUNEdnMj+eCdwZTmPTv7wDWNyZLHVCp8lwE/87qp3erjFpj7TOsDCimwVQUxcY1RD0CtK4x84Jsv9tVXC1/mEsBcODIZovPPZ//M6Y+0qo+zkPmxYDU1qR2X60bWFCUXBmczhw1/1PVLDIukPixemFUmXCuntvbbrIofF3NN0XXSC0V6qidHhZ4iFQuWjC+cCKyhSgCT9WOuokRlJdyUFUfAJ6BTJ3qlbta0mMmsDc5h4ceiyCQWMmardJqtcNHfP9JAleC6ytgKy92QNbb8Wly9uiMJnbYlBe+OZD2eMCPHJYknqw+vTOdpf5N748f6rdPiqhqc5bkBkdncSDx4XjBlIuEeYq7LKJDHcuVdd+r2+g5Y03U6mOK7P8n2h2+Lq1+d5bDSpFFMXGPUB1DrCiMd+D5ebC/ZJ9/rmGr0aVeX0nG5tJVNENSkaiVTmXB5ixPDAOHZpg3dfD8u03puLuQyfpEH03xLl6z2r9NiJrs2WNd+Lz9gJUufyJfA64FZI3t0XdqSoxeKlMnwo8FVmu9bHYRTgag7pH2CncUPAcjnjXTLluvIyIdZy8MP6XROQpwa2unjqvp+S29A8nlvx3LBg8/385UDzMJTAiEiIl0X6j6L3tF02j0nNTc76vNgtpuyfFUQWSOCsnuE6AHJlNjiGuMYA2pdYWQD3+eLvYAj++RzXna3SVa31WR7Esf3+Xwwv0a/VjLZCXdgQPwoWNfntSkJnTajVWgdKwP0EjJVAiXTL6wNC89Sy9NXtZIPlbXB+pi5mZKsD0PSTKQP5OlwC4OtsHYQznLPPSN1eOlvVcpTPTnPVL3cvnHcEy8A7VZ0i0mBhyQW63V6vMunD7rI1UIESwDYLA5Z83qFu7vF37F2g+r7rRws5CVzwXnw+kCetiXEY8Ucsryu1XWibzZ4jAmJd51VrkjnDb9iZ2McvtxXpPMuJRrZ/jCHvZl8S8pO4VfNIIJAtZi4xqgHoNYVRjLwKy+2H/1VvxZXV0YsTtR78eFhOjyFLyTumlw8SIo5hwU9RvU713nFUfX3q19f1CYVaTPTkiPz3cWpnFoEu6VYLWZ+1wbrY96UdVvlqhsWSdOZ+TWr9bOjuUjrlkmSDOc5rUR5siPLJPpOYu/aO3HuaUN33ja2glhcWf1cLBruMdK+gdZBy3K1UHTk9XvSa77C3RJytn7eb+Wjaz9Omnm1NNdcAYyCMSZU5noZ44SXe5H1OZtazKrrzeNdOdezcT7PoO4RonaHgZi4xqgHoNYVRjLwfbzYXsERonLb+/NGvnDrh6nUSHQX+bCqKd6q1zEzL8d3GBGt1nszidLiVI7WZQ2/VNXri9okHaAD0Kbrc/S9i+VucF22aCNJsgS5E722o/DQ1gbWhkXSdCb6mrRlXnVRzGTocLM4Va/5ftnGksc9bejOK6uDSJu7fTBHYQCXxABgKXOY/7YGMfIu73yFh4cN3r1kifH/TgksP++3Mg9VrcTDsMCai3leGYUeybotxcs4oSS7FsBIMkvQDqF7hB81g8Bp0tiIiWuMegBqXWEUA1/16I+3yxdlbjGLZyrZdLpqOXL+zuU76WNysc5hIrmpMtgLalhyjLpuWAqdBP5gyn3053V9UZtULK7VmV+SDVh5jCxBtlpvIlobRjpYwnRmHrHzvubbn1NUNE06Y8/8lqI7Ml3inopFovVLQtQO5RBmr6x5WxMZKotcLUT+nBLGciklOIWIIvNd8prLrF2mnHxJ1UnThzSYqbHrvO+neuTn+jJAR5GkcRgWflVpg+TDSFLWDM1nkZFEqFolW2dvb+Tpb2PiGqMegFpXGEXmrI7motJEwgqOkPHVkzm6NLJusSOGE9BpFoq05OQ+0m/vJerrU55knHMYT26qqpHq8XvhIiVaMLlEQC5iWtbimpDwcSXA7hgnwwYq91bqMxQVfnm93JH6xt5ibVKjSz6ojb1F4Vfa0e/LOiQs2az0DujxrpzyPVUHR1iClwo+gi8hQz3IVgnfbPjz51TxMxW6titGWRWLcnOZ2VwZ/XphsJCMk6bkWMmip+riot87wLxv/cKLlMerl8VVSXbNh/XTGjwqKpzMsD6juaJBTFxj1ANQ6wrDHvjFohy5KQM0hBTNhlv7k2dFdVpmZXfmPKktsy6evqgMWHOYU27qtIzOncOk58ABwYLpQQREbhgiH1dem+YhTyV4BGM4+0/EBlh/a201zB6cRpQ1jQ63tNGqPj160kpEpT65BXL9Er7/norfrlIxmbvMd4OcSYe1YEu21eoGYhY/WrJ+/EyZe0RJ9mv97ZPLxHPZPORd6W69bo3rDSGrBe3HR1NQVDdiX8cS7gkaUAOLK+SIq9DqqrJRiBAxcY1RD0CQH99yyy0EgDo7O6V/E/bAN9fCn0BuJ+7cfTefLF7crSRMxd+SVZdKAIAIQecwr99v6PZYMCXPFp0+XaL2SbVJVV9L1W+U9d9gW5FVBfZVkM/LnyKILK5KbhYqRcIloxxKFJDEwJB5CArpV333oaXNoRiKPVil2b/W1MxeGxVzLuu/R1dqJ9cSaLbTy43Ba6xAnYz6KSwFA3u/SWyQvDZTzPuTj6fwlFgMmlc5IGLiGqMeAL8//N3vfkenn346veMd7xhV4moejclalZyL0xO3F6V+NxNFZSJgrctv3nUeWHPYaRlDdkfmLJs7B/ZLmItSKan7X5zKua/PU3eqSAYtTtl98mzz8vAw0fLlRBdcQPSJTxA98ghfzYHVDTKmMEb0OMu1RCQt5FvgnEZ4muxiypORAuSthdKkwTlGVUmlXwtq0AXbh4+l9Z0qK7RZH9bpexeL/eTNInTNlWyzdXMoOz91NBcpnw/P310KnLESJWn1krGznlTx0npzfTtY497H+BL2NWsyCTrBBERMXGPUA+DnRwcOHKCzzjqLHn30UZo5c+aoEld92MiCI0tczcnDXG9kj2QXIEfjMExHkfSlHyi7qBR6itLzkaekkodZkBXxvLFXrp0yRS8UmQZPZzNZ+okHUxnalLWkhWUJn7e2cmWhmN0ga2Kq6HWW+gx/O1FaTiePC6KY5OTVvOC7MkdVwLmOygZmmX7Zzs/s/zYC/vQBBZcM1iAzRX5VLahBFmwPwixybenuFnSwo80buvO0LSn2LbUWoXXNh5VYdqOyEDmVg5PwYn786oGFWMrQ6JXGNrr9/QO01aEpewANdGRiI38sB7wXGVmu6pgIOplEhJi4xqgHwM+PLrnkEurq6iIi8iSur732Gu3bt69atmzZEu7AlyQjJYBeRgstQp897aiCNSaIj6tq1LrUHGUGGHV2sS0XCsep5jwp7Q6RSilbzlin9FLuE14RJBKyUJqmoD5RMXuoHPkODPC7wvoIRPyLVR/LJ/pwi3sx5VngD5yY5m60ygDtRJraMeCq4yjsApOm1Zk5Lnk3xVt8u7trf+QpIJ9lQTBh9dl5EHRT21NWpzSZNMYMFwrz0jgMUyd6KY8LpH+jacbteJ16O3VGA3Mn61iRTHQRRTE2a5y/NzZ66FM57kXyPj7WWOT+2TZlqkT21RgxcY1RD4DqD1atWkVve9vb6PDhw0TkTVyXLVtGAFwltIEvaZlwLt6H0o5AI67/lbqPK8sZXzWjk+ccJbvjl3BBsM6T0u4Q2az4uMyiaUvEPqWXcp+QyfyUTnvKQmka0fwWyXurmD1kj1Kvvtq9wLMewQA7ULr6jHn1OaWNVvXZ+9WqC1oo2MX2hxu9iWsCuquOcRjmpnGVWju9Ft/+/vCOPGWtsYx3ppxpoyvS/MAZ2+vjUKIwA/WGXxWf+vAsusJ+1HU63MJ3FTGveRuuc20yVPRSvV5j3uMLhTv5DHQSnnhNmCB9Hc8blT2tKsi5k+T7dW8+6ieyr4aIiWuMegBUvrx582aaOnUqPfnkk9XPxorFVWiN5Ppf2a0xsqSuB1kX4ZiNgqF5qnBcyZ2jKu1V8g3jnEs650mlQAUv8mxhZazHFGbw0MblBc+vJWCQgbKkpTjkYGhuleYwVA3q8TxNDODb6dVm4dqpGwu5dLa4IFA9UnWQXNWEGazqPtbor5+9uuGxLr5OcwkarcEcJpFjfcaz/OZy7Hvy2oidltFJLxSDbTwUA51CV8fwKgr6vFekjcxgXu4knu7aYUnARYSYuMaoB0Dly/fddx8BoGQyWS0ASNM0SiaTpEtMXqEPfB9RnsyVgzGjOINyvEid4SKQoQR05hHvcGN6pF6JRYU5R1XuVzmggRNhwTuelg5U0HW+s5zl+yxroqpKg6hsuqBH6qvfP5/jNwr3vckMLdECrzoMh4flY5ekThMlTcbL0eWr3by1c12WMahYpeJLHDiSTdgJYsha1ZcsGbFO+h3Hpk6pM2jL9X5X+mRjb5HpxvES2mg+7hH62zs/Z81lM1E0BP2LxWrSh1zOOyNuUEk/661uyuarBgKv/huCXFCoyI1Jmfwq6PPOg9tXn+UCIzwgqGnEnDpi4hqjHgCVL+/fv582bdpkK+95z3vo4osvpk2bNkldI5KB7zPK0zU5VSLbF0plzuKTT57fZtXS5zj6ZkWtc+cov2ZADsvgzZPSmb4kj7ZYlq0wLa4vfEKOuPLu7SW00fpr81z1LJYhJaSm2x6R11Du71c4TSzI9e/LaPGV+pi1dubzCmlmneoUQSLZuJ0gJsNhWNX9jGNr0Fa1Hxmb523JDHWg3+W20YleqXp+hE+45jLW+C9njGBIL9fT0CT9+nXqaB6J6GfONf39tLG3qJwM4m+XZ0cCGC2fl+CDuCrq805v1emJ24u0fomx8dCHFTdjscU1RgxPIOgFRltVoArWGYykbJM5OckKcrMm/tcaUnQDsnQChuX8NgsFeqrHWzLHNUcpZm4pQ7yAi+ZJq6vDxt4i+xqSE61eKDITJ3gK5Dc3y13/kYKSdZSVEpMXiMI73uvqUlsDJYeh0AOjtZXosssk17aCLt1/ojzqvFIo2IeC+f743pDwyA/LRBXSAh/kwMY9jtkX8Tq67+2tJPxgNMLUEJ7n2Nh+HUukGvd1LOHIPvHb432fgucnsVnY0M3K7NVKNyBLC5EzLPaVa1inO+8TrxF3K5ZSidMXWHX8+NnkiPZiw8OGdXvJEuP/h4clBmTs4xojxjFEXIncC1xBboduTk6yE1NPj0EK9GVZFzkeSsgRBSoWpRbNlpbKhGZCYfYsmUdwHkEGXm1IpYyuZM6VCkdbLGviregWB1xce61UcBbpeujWUSuPCsKdZIuVY/FUClRKLkdE558v9WWn5q5MqS7Klc4wN2Ljqps3PpETdrqMxpjsrkF0pFpp9+NduarSiN++5kqXCe7XJFrjMEzbknz3nzLcue6XNfVKNez7uNxtXfVoTygZpnhdPiCf2Ys13cmceJnDKAmdisuKdGdDF1NFQGkcKkx1zsuw9mLd3W43o2RSTX5tNBAT1xj1ANS6wpoOfEVmqORexHF2kvY9rSyoMl4O01stiQUKBcPHVYKNbU22uXU3GZD1tGBaDxQsrmZdspl+CBgJxxd9x9KofN7oL6s19bSM7slzWBZYzvplG17TW3Waxfid6uJmrUPW8u9VVCyua28q+mo3y69vMzJ0K7o5BEO28UXBO6YQmMgjUQxCPJRopuXo8v0c5yLv0gaVKbLH/ht7iyMbp1eHhQ7WZYB0JNjH+h6FF6znJ/2t84UxlBLYv7MSZ4v3lmsK57n6sJKEnJbR6VBKMe0xhyD63ag63+/ubvH3q+R1lLNksRAT1xj1ANS6wppaXEVOitZSYWTSp4+FEJiFZUFlkS1z4WT6mlYskM6ABtOH62voolkoGlmwJCGjrsWczyXF3ae36iNapsM6bewt0qYLBM50zr6SjR7PuxMxlJubadOyfu7lWX3sFI6vPi7rGMtmmaRNJi+5qE/DsORmMvI+ruWWFpre6o+osa1nRsnhIle/ygbYPHV9Hx1M8a2QJaASnOTjSJUXYRPwOba1ER3JDdBwU4tSR8oe+7sIIYcBmVbeg5joKxKfJ5Qf2OKqoHJhvVXWFM7baPpus7VwCGJ/v3cwpqhdZpIXr2skkxa3AZngRdnvhYCYuMaoB6DWFdbUx1XkpOhcdTQjK5CUe5EkIRBfRLe1vdwqsloxrgFwg7yUAnwtk55eKFLhYV3oGszkA5zNgfX4zvzzhm4Jhuws5irmNUELCEkZoG82dLv+zCdf9qPHqpXdo+0yvoLWRW5+i3FUbb0fP0eSzpJOyydc2NyhriogZS2H4bvYgywtgHEcf1GznPuOrBXSIGkKR6qS5mzrc/QiSbax7cMvRfZenYRQ14leuqibSloI0haWMht2abmeHjW9Ui5xUsgG5uS+fkMYVBQf1i/hEz+JvY7nBlhGscEsvb3sLmQ2rIYZtmLiGqMegFpXGKmqAG8WyOeNLWyLwBJSmXRNkWiWe1ESlSP7JZIWElYdzgWVo8lagofYdiXIa92yAn36ZPuCysiEyu83x6R3uEXO0uQyqjAsnc7jO5Mkqkp5bewtehsRJAhJGaB2DFSfrRf5sh5dbsrKkxKRr6BJgn75zi63ZU7R8u811GRTvhZ6isrXV8kiZ/ouapoRTS5joV+EPqnrfw1drjSrwiNVRR/xnUh7WuPb2iz3pdCJVh9Xae1kxqs7DsP0fVzuLc4vWXbA7jJhe9eD+F9K9v38Fvb77ieEQXacZtHDDUCV2evIbICLRfmlY8kSxg3LbtQj9IWNiWuMegBqXWFkOq6iWSCdVgrUYm1ir2REqSoX54LqRaYlCisaXGreEvgPylgMWW5sxQLfMiVroXNO+ubC3tHstkzaKy9KXfNw48ixuOyidmFzwUXKZYrTV5Dp9sF4cF6Wf9mShJHRSegPzZEq8yoqGrylSnBR1XWFa6E3ynJ00fdwmXQfJ2C4nkgdlfowZ7MUAcrQ6LGu/Eh1irsNp2VeRTvZ+er6ebdky7Zkxu0nL+t/6SRew8N0KM1vp0m8N1wryoVrv7yXAfi0jE7lVoVAQYa10uvRymyAtybbSB/WpS2uD1zuYUkNSQ5OFTFxjVEPQK0rDH3gyy4YHR1y37NIY5lz7qZsXioYijcxHm5I0cbbC3ZNv3xeOnBGVHj+aNV5a5ixa/eY9LyiiwG2G5uIE6j6mpmL+K3olhM7VyAkeqFIxSIZR4MS33+6XdIXV/BsuPqXrvGiUTnDt/zLFKf8URmaa/yaou/6QN5XMJgv30HroGGQH9nUpawxKogHsg//IG4+1uIkBoqEmBVMNA95T+sx61mFqYfMGo/SEmVWMNPrZuiOydd6KC0YaXhlCZeUATifr27IneNLmFFRcmpR8f2V8XFtT3DWG2vbRknvNSauMeoBqHWFoQ982QWjsVHqe2b0exUhhXjPRHGEb8k4TClcl/fnucjT4RYG6eNlupK8tinEoHJ0p5ol6yW0cX18yyyTsorFq6+PdJ0MK53E90vX+yOupjXcj0VsXZZt+fcqLKvugRPTtDvB94fu7vZWHHMWz9TArH7sy7nfrWKRSp1suSJeMb6rsYPnLGD13/RW3Tg5Cen9q1YsOf6e7uhhSm9Vh3S/mBCyqlmOLrU2q967qvVOeJojWadCVjUpA3DefWImm5LY69Gqqi2IVAUS0OmVBglLap+cK03YGbZi4hqjHoBaVzhqFleAhptauFHIrOh31euLygLkqn6ygV0OKouAyCrK9SdVWLQWcqy5gEF0nGSntdX4zOpDalr+ZANQsuhxaIHKLS6k69Kbkz9c1kuZjJxw/OHGFlrRcL3yszmUblN2SXD2vVM/NpsVE1mxnx2qQVJ+5Z6cxczPLutXubG36Hp98/2GO4PqMXcPssxhUL2uwP1vHvJSKUalijVwUDJwKYjSkXOfnoBOO6B4ctPWRtTern6vMtY7z9McyboUs6p5GYB13XBleuT6It3/TsmNaOV+vR6trC+5tf8+e61O5yTcCVC+dZHktWR9DmKLa4xjEKh1hZH4uEpmyFqOLk8HepcxL4wQb4xYL6UnOYgtAmWBH6qXhU+WaHQ0F5Vu0zqxt2OAdsDuv3sUSSERPZgeIeK+pHeuvlrqN4vQV/0n17eQ0U8y/WZN+mAuprIuCdYyC0UmITOv2ddHtHz5yNBXCTRTbAq3FApElM/TwZNbPfvkJbTRqj77zeTzau+DtVjdMPxkIb4y7Q4mtLdZsi1O9wfGubV5JP1ke9ZIXKL7VzBy7qOl35OmJmPQmJWJ8rryioz1LuysHNaH5jPgyLlR8KNHK3JJSECnrQlxRi/Tx5XZIID2N2XoyL15+fWmr29UMmzFxDVGPQC1rjCSgS959N2DLA3BfSa6E2kbCbS98wEnYidhkJ00hRHC6bQwgEcl4lvU5od+pivHjmka0coTu5ltN5USeHnON2VHnoEvsXMFnUjrR3ORZ44LFnH1IrMvoc1QILBCMZLdOl68DCbmgipLAEWuJarF7PpiQa/40rqfrTOq2oRJLlVdSJz3UUma5qu7i4URV4WhhH2gv4QMDSEtICOgVxtb7GTQfCACQrwZGboinfcd8O20/kn3X1eX/UKywaq2DvMYjETSxMsrQIv5Ox9kzGp5N0+AslCzuFqvxbKUZ7NyGb2KRRIfBWga6TfJrWV6oTgqGbZi4hqjHoBaVxjJwNd1oZOeKWsjEktnWS+LRQqUzJyl6SlLKoXWvUxGmOJ0oQI5Fk20F19s/4mM6Hc7+j2DL1w5w1MpomyW9GG92tW+LK4eprYywLQ6qsh0uYlqxqZTelpGd6+rkmOINV6cRi5ejo3FKXmNTJVh7CWobr29eZyMRs40nkQj5NJP0J7zGdr4ha5ToUcuk5lN5L7ffZ9zkeduVnjR6OaR9JMdWeb3rKlNZXmF85kPDIwMJd9JATzmTFvhEUZdNwhwT49RCgVpQsyeeyTHgeTxt3U6YPl+81PsgkuQWe+fydW9Mnqt6vM+CjjcnBG6L5njv1gQbJQizLAVE9cY9QDUusJIExAwiIFJVg3rifxiaFvYfAZTsSKH/QS1iCZv1rw1v6UodY0eZIUT7QUXjPxJJruUir/d93E5vdbg9mPb0J2v+gJLa1s6HUG5SQjc7hXm81DVvzR9ca2pYYVGDokxZO17kzA+1TOyQoq0xmWj5U3fUJnCe+YsImrenpPoJqEz+8Vc7FXeB6tclrXvbe9pRjxGGa+Q7R6sXM7Y0EgS14r7zhXpvITbhpFB7KJmu9IIb1NivaUEdOpoLtLt/5hz+ILzLcMH0xwrZT7v2edVRQpGhjom8U2lPAnxoXQbXZUacI0t13zAK5IBR+bmSFbRw/ZsB+Qkuaz1WN9b1sZJNhi0B1lP662tC+LMWTGOM6DWFUY68BkL10toox7IHb84j1FdwtuSvrQ/mLyEZgmsPSoTqczk7Zq3hsUWPmtgl2iivf12cXudFkIV6xmTKFbY34Zr++nCdIHuRYfQtYCbFa2hwR2o1dZmc0Uwi+zYcJbl6HJe3tvI0d3t0sLRkaD7cb5nmt9DaTYBs0Wje1h1WdH4vOL1zDd022+Wx2PSaXa/WBd73jGrc4w4LfU70Ezt6BcewbKs2KITZ1039j6Nk/3rDs+G/DH84RZj58Eaxs7+5G0kRtQ33BbMEozMaFxC4+Ha8BLa3K4NAsIrtQHs7q5apgs9xuZMLxSVtLZlkMsF0LhVCGqSjcsr9cmfinhZb0OOuZJGTFxj1ANQ6wojH/gWFrex1yADsn5g5jEqd2GTnFjXZYueGpzt6HcfmasUL7kYjnahGTx0SQM/sGsminRlQ46OPFKk018nju63Wqv9+iu6G5EQ/91kiV5WzMZGw7ev0j/OBca0qPlpYxmgyxrzLhdHLiSJlewmgbUw6gPeFjSZIC3v4D77CyKTuI71mjqfhXOh3o2TadWky+hr6OTKZZUB0q+5VlqXWNb9L4jeq7QPJUYsfBfiHqEbTjsGhK4HLL1j1/zCi8yvzJmPd+XoHBRoNgq2dtj6TNcN+RCPe/J8f3luB8r5pvkoFgNo3CrKSEm5m0o6X3c0F22nFwvRR53opUXo47sj1QgxcY1RD0CtK6zlwDcXR5XAFeHCFlDyxlp8T6iaZphiPPJT5/Nin0OWliCLPOzU5I7+Z6IYqRB6tWSzUkkUbP3FyDqU9OkiYJZSRfJKagWRTPjgJQHmRTqLRZIOVOxELy1Ajj7W6CZJKn6TQRL4OBf7BIwgr12wk5ejSIr9ESXLTBTl3f+6unyPURXiyrsPq4uD1ybXOn5mokjLwdHFFUxu0s8xAKF3D1bLQPB6l30EHOm6vO+3sH2SELny2DrZY/2wJh/hnb5E5cPqhZi4xqgHoNYVRj3wWYEMXv6SVkLgubB5WDKdWW7MtjjVZ3xZJ0VmLfNv2SyV+nLU0VzkugJompEKce1NRVqcMj5vxwA3eE2mbQuQowR02tsoDo4Kkku9DCP7TrVjZX/LyPN+W6M/FwFnWZcteg9Kybb+/uO9Ut/jKQPkcuRLvs3pByo9Nvv6AifwGRgYyXociguNoDzVk5OzVOl6oKx2s1EI7MdutaTK9of5fkvrHzNOp7yqufeCkE5VTIumbPyAz4CjdVnJAVop5QAyUvm82xjd2upOhCCjBJDPGzrJzPeBQeJr5eYaE9cY9QDUusIoBz5vx9vdbZ0ENNdE5co77lGHyJLJy3zjNIT5sk5mMkopjniBKaxdvI5EIFJpkqnHuvgLURmgn+CiwIveumxRmaDZMqLl84Hu1VoWIkf9/R6LhmRbfzB5idT3suhhkoxikXzJt/n2VW5poce6vP1lAfbJq/V99e2LqFJkrWg+JfCsG2CRPrD89Ty0jx3F3JxKXZ+RzUIUyGaW0E5VzBfFy9KaShkuBH6ZmG4kfJEL/qsoi2SzygxQpHKVhD4yZ5nWFJESQMVtopxKSUmDeVp6Q0RMXGPUA1DrCqMa+B7yeNTfT7Qpm6eDKcGE4QFdN1JGOv2OTEsHM9d4JVLe2S45wfgM/dvEAv3oI4ZFRH9EPujDvIbTLzJsq1YZhg6uTXc07xZ3fxkt1I7+UFQVFiKnbElZnMqN+OiFkMLXLDNRdOUeb262udZKEyHZ7GIEg2S0o79qTZ/fUjSC8nzKt1lJl/Qz8kiEYS2sCH5rE8MgRKIxXU4miQYG5CxTPqzWvHfNuUGMsphjwX//8f2ozWKMjVbhxk9ai1V2g1AoyE3OvAc7MMB0nXC2cSfS9FqDwzAgwQBFUwpzDGQyxN3tKuZ5NuMpZNU8giImrjHqAah1hVEMfBU/O31Yp429RVq/xCCDw6/q0kcsm7J8SSgeISxr7MWAJ3ovWkD8+GuxCUl4i6VJXJMVNwtTwsck+Kxgk6DWqJmVAAVR5iNnmVXxX5YlvCPWLm/3EtGlMhnviH+3j6uMdYhBDpqbjQWRdxwp2bf2Z+Tx/DUjK1CS0w+sk1fW+xpGYB8vgMv8mylX5Xo+zsXdh8V11yS39B1gEIrZKNAupCKzJpcBegWNVR/XINeSGddeWs1ljPS3azBY2ZTsBiGVEjMwkclRQASHcDJ9D5fRIvRVJaiYKbI9GCBvuHDfIc2QGNuUNWStigXdOBHy4Ve9OMVWIODJ1gVFTFxj1ANQ6wqjGPiy6wwr17vTWsbdYFd9W90TvapOrBcpcGbyMotskBmrzEYh0uCpWShKBfpbJ3WXJSIhdlcwEghkqv24KasWRa9p8uS/DAhkhrwtU2Yx170N3fmqhVJ0LRVSz+2r7m5ly41ZrAkK5iIvrctrbgxY9+58n1jva5CxWYLhqjMf90gFMVlJGbONMlbrTMawBFZ2vD+5W7yBCaJgIVtMK3wYOtGiDGvSSVSaHWn32tpIHzBcslb16fTc4l75NnF8Oh/rMuZlJuHktcvVb63C+dtLzYDFv2VTMLfDrWerUkY0X93XZ2WsC4qYuMaoB6DWFUYx8H2c7AnnR95CJsoGJbsYyIiTW8mZtYzDMOnwkIrilF1I0XJ0hddRjrK6I0d9fWRLEeuVacv690JPsXqkx+pn83NXVimegCjYBFNWt9UU6vfSU5QdU6ZcldNVxVi8+m39FHQxI8Dwo9MN37qFyEm7ITgJyyL0Sf3u8a4cZTL2Zzq/pWhYmyXeVy/3BKaFGaZWqfGMZQmV8x6ZvCSfr/i/cx6og43LbJ7/E9cq+VabVn/Z35h9wdtwlRUs8KIMayrBe1QsUqnPON1aerWRQtqXCwXDpzOsEyTp33NcFoJsxESnBF79cTCVkSLHq/rCM7nGxDVGPQC1rnA0La4+5shQK1AJnmBZPIJapaK0+DjbK5Npy1pMq8BjXWwXCpYVumpJqKjGOzPvmH615key/nkvodVGsmVS3cp8p1g0jgW9SKppOfMjq1QtLS3VFLoj9y6nrOFrzBWLpA/kDUF96+eMIwze63QruoXk9DZcSztgt+JZNxGqes2uWyjoVaL1w8uK9PUJ17ksuKVE0rBoO+DlruSVVpinz/qXOQzdOkERWfIOptvob5fJbdw29hZdSih+xoTT8M91qZK8P9OnU6kdYRWOywLLQC87Fv0EiZYr1hWVZxkWYuIaox6AWlcYpY+rD5c+r3mXiOQznngVleAJ1uIa1A/QsOAEUw9wX9NNeFRE9J2bhGJxxC8wix7Koodmo+AiVC0t7pO7TcsGXMTGSpZlFzpeWlQeOZUl6bmcfayK+wn0PVwW6NkMLC7aPuJnqOK7PkhZtdraDAuvKDrScczrfF9F7jNlgP4T1wqfgcrzZW0K58JtDecJ/jtl70zwXItlNIOdf9uabBvJTiapzeu8R7OvLpsw0lcJ6LQtmXH7n1qfV+WF7O9nf8UzeM+hR6o0ljzK4tTIvBhawhOVwvJ90Q1pQeuYjJJUm2NDdl0q9aklUxAhJq4x6gGodYVRqwqw/Oz8zhG5nHHdjuai1A/41hSD3CUVJrRO9Pq3ftWosIiorG+XMyOP6bPmdDfgFVcq8crRrogsB9k0sMjpLpxMj2I2h+AYxWrxNTdC+bxVW1htLKkU2fs4mDLSevLelf+sWEK5bbpWnLmqDI0Ot7QZqgeM91WG0BxFktoxILxlOatyxpUZimcN5bsGaVTOsH0eWa7F81vk3tutFyymZ/5lCT23uJf0V4dHLjo87HbG9/HszTKver98p2R5C7LbHaEMjdZ15l3vcRjzl3XTMSrzoXOnnc8bCQEs35HxNVYl70YcRYpmo1BVDJBWVgnRyTUmrjHqAah1hbXWcW1r82WwIMD4nX1hZU9CZWhUTqeZzNlKnJJJCWuFYwJUIYVhWlJlCsvXU8XqZc3cylokedY110mtZGYq2RzyLLcHvxJiOhL0NXSOyFVVoCrn5afwAmyc/Vp4WLdlFLP+7YSK0oFwbMnsNGCksrQaq8znruIP6OVbPBdmsA7rPYTLDcUzMEdUOITAqczk68TG6mLhw1VJFFwFEFu6r6XFUKWQrHIu3K4hW5N8/+9AFlJNo4Np+8lOGNJ6vkux6BGwq9EazOG6vqj6OpdgnASZ928mkSln+NZzZ2rmMBAT1xj1ANS6wlpnzrJmB5W1vmqa8X22bxYj4ME0GzIYGIvcffkfeUe27AnQrcPKD2CKZJJubibq6yP94QI9cVuB7r0gR1/4INuPU3ZxMrMY8VQIWJbBrYkMrb+WIfkgubB7ZTQqwfCL/dDMYSVroHSxkpEQIgpFwYIyUl3OZm3odpOZoYT/DFLOshA510mrrhM91SN55Cl5X491ud/DnUgzA2ECPVfZfPZ+fOSt84rCWJGVaQOI2jFAw02OTUdlMJhVevlur+ozJtzHu3J0Dgoua7b1uyobFFZfbMqyk6kwVTjC9htz9vPnrqfhKfwEAeYczXN98Vuv05ixKZuvymvZ6vCRIlcGMXGNUQ9ArSscrYHPsyg5J1fzfWdZaVlk6nCLPYGBPqxTR7M4SCeZlI+sdS5ECeg0VFmEeZO+aDJ9Gc3ympKWyU8QvO9rcRLluRdp4jKznEku7AuQk5Kb2owMXdKQp3Q65ONI62ISdkSh5Rk7rTMyzZoHtqxQmJsh0wpoZu2tQrEvvKyJ2SzZdrCF6wvhbT6sRfYI1q8TvnksXZA7KbAqLHh9nRssZiGJMr7bZlKrSxq8v+t9cgXaiwb3vFg5mjHn1oUSPubljOU4J0D63iiKyjj0MmZUlVVEmbhCRExcY9QDUOsKR3PgsyxKzsnVfN95XMhpgVh6tf0YxroGy8hBqcoUyRIplmVJmZRUOiPvLZcqvThZfcRYfEXFR7ZqwJQkPlZxfdGmwbo4rO4IOQDEvP/hYaX0vX6KTApPmT6XKi0tQmLm9FHNZi0vTTUtp1xdIv9NwE2MN/bKjQ/ZIvJx5SJAUghavlwox2eWIaRCe97lhinc4DQrcWptJbrxbfLBmF4nRmUYfuGzUBzJkGcezQhS1Drn2umt+ghv65OTdJMpQl/vGhXrHGhTVpHNpBMAMXGNUQ9ArSsctYGfNwXgHRMRx4onawRyRrebhFc20lw1YEj2+7tgl4ZiTbiuCTiTceXp1nVjcVKZW4XHd5bjK9bmQMVHtno5ycxU1o3DOAzTDrQIA3BeQhtd1Cxn7VIuhUKoxFWGZFiLdaFXSTXrKiYRr6gKiKz9zraYxCKfNwOG5OrkqT5Yi9UYGkQVhBV0V81nr0oOfCaFIICokrdeRJpkAtiA4KcIKtn4VE+MnMGbmQzRk8vYpwGi8W2bakI63agH0mot81uKUfFTLmLiGqMegFpXOCoDXyUnbOX7eqFIV07hH/dbi3WRLBbV5KBUJXxkv9+F5XQzrqdfYYbwWLwM0D400I8uLTAXYr9zPtOq6Ti+Yl1blcgnoNOcpiI9/eGuik8ZPzjOegnZfpyNAu1v8rZ2KReeSKbPIkMEhM/GRx0uP7r+fion5DNXmYZn89WUEeh3Zk/jFZv7qeQgljmlsBWJPPYuvPoqUVNTaM+X9T1zrJv7Iud+bmFIMlIzUVRSSVHZJJnznSoxtpbqtD7s7apRT4RUtjzeFZ7MlSxi4hqjHoBaVxjVwLeelBQKtmyMRh5omcmgUPA8kmIV6yKpD+u0LSHKsuW2QKgIw8tkGBKlveSV26Zkmbv3IDFECeg0C0VanMoZz8BRAcv1T4XIswiY895ZmakS0JUIsuh403cJmbjK9BcQTCFhKGEP5LFpjRIpu2wARL29I2NFxV3By8/V5n5azXzHf8d2Is3IkJahG7CM7kUH29omEwBjnZiy2cj9LUsAvdLYRsWCzjthl5bn8ioLkItMS3Uh+ggIps1rGwsq/k4hPAOZ7+1FY7A5JcxcrpKIiWuMegBqXWEUA9/rBE42Pz01NHAmInFuetv8Iam9ZZ1oeUfrvHr53/d/nDWEtJE5yIEwTtlEp6pOVQFZIt+OAY5V2yjL0SXMTCWb+jWLHpqJIt2G68IhrooBN2GVBcj5llMzdYjHYdi2ATD1JPv7jXGyfom6Tu6cOcb/qx5fL+T4uXLTypuuQgK5Oqv7RA+yclZpUR77IK4BQYtlUnK5P76qrgvLKj3IRqalugMtgbWXzVI1LPjVRYyoLEcXM0BOpEhAiEbmShYxcY1RD0CtKwx74PPklKxlVgiTK+tIyrVmKezqN1yds60dLOshS0pL9H2VvOas8sj1RVf/yvq4ptPiNVp0quoM/PUi8iPi3uJnxSe3xme7tLTQjUK1bz373plxIYp0b5yicqxruydNvGkDRjjQjbhJui3Oj1Utd7NQdHWdyACq64Z0kDM7Fusd82WVdlq/ZCamKItIqiuE3ajpsjGuovMbtmKD+a7LbjBFFtdqIKDk8ZFfxRZVo8FMFOnei9gSireiWzgH6gPhKwbIICauMeoBqHWFYQ58L9dVs4QpVG1OkK5FUrYxZikWaWDA3U6vfPe87wcKsKmUhz7BXuyc7WQVk4vxjBpep6rOwF8RkZclYDvQIrRalNNppoC43yKykhBA1NZG+kC+av3alM2P+Ik6rhNGe8xrmT6hfo51D7ewXS2cX21HPx1FQrotzj8raXxW0omar5v5HixOWaLQLbAaPq3fLS4rUrGg06c/PVKFb4UFm7+Q4lwQRREdI4egIWwW02UndKkxjGQ6U3GlYhXVIC2+rzzvb2qk1ZoF6yd3G0LjhR65VNLmHDgKXgJEFBPXGPUB1LrCMAe+iuEgiG+ftZhHUi6ZPJXGWMy0/f1UzaglQ1p531uOrsALxSOfK7gkVbxOO9PpkX6QiYE7LWMEvjllW3jSWKx7DdWvbtkyKgkCilSKvqTTlUmIWlqIurqIikUb2TLLFWl3ykiRxcdPu8wofFly+Nzi3urzGexku1qwJI5U2sJ61l6bS5M0mH61PCuq1bzPM3xaN1Kmn61KH7mKOZYLBaKLLw5vfPopLqHcABOnRzHnwzDmH9GYYSdsMRRhvCTApIO0NK2yUXOPeZMwssjky5D3W3a+w/ubjLFqPhLnnOd0zzHnQNncF2EjJq4x6gGodYVhDnxVw8Fc5F0yUaql0FNky+SpNMZhdlx/nZx49w3Iutpv5sXeoTB5OosZnHK42d6GQ+kMzeMsCgno9P1PFA2poUqHeJFPpt9ghWSwTs6dk/gJCWPSDtWvLswgKTNlppX8Dw9XMwvNYmxIzKQY67JFero9moAtqwKDjPVqVV9lcHuktDT9QlUslCJfxFvRLSTnO5GmeciP7Ps8WKk+kJcSE7n77pHPfG2KWlqo9JlOOjKpMdizamwkOuGE4M/cJpLrRr5fp21JcSIA2bpUN0V+xy8zk14lOFDWdbUapCXQ033s6n4C+Jtmox1236nhKSdL34tbknBkrF6RlpNQrN7LKCAmrjHqAah1haNlcTWLbL561oTDEh03Ax8KPZKNcS4q+XxFS9b+PSs5mIu8K8e6/Xv+F4Wyo/DaYP0TU04pk6HHury/J4rMtq4pzIVKy9C9F+XpJ3frtDUhJmAvo8Xz3gkIl7iai6GphZvNMhUqnEfvSejU1kb0zL91hdcWS1ENBCwWyRKJz76mSXJV3yeeL6LoRKQM0E9wkY30Fwve5v3DLXKpTxcvHvnvKAmYsCTEbhYqpXR3H1eL3nzHuCmsoSa9Zbp+GBuY1khkpazSWKzgwK4uuUtVrZSCY6TDLXwFmaCndty+0TQijtuScw4WxQLWAjFxjVEPQK0rjMLHVSUGQvZIkjWZrzyx22YsdfrOefrRNjcbFjjLDYgyBpU1jYYb056LSVjElX1tuw8ZNx2r49iuHQPyi6BlNs7njfSRIh3cDd152tAtJmDzcY93QFUySXTjjb77zk9hyWqZZHanpnbk+Aqm0Ga0Kvv/8XznbNZMyV1hFnLEv1Spg0UkvbU63b/9Rrtc+7xks/htGcXAqoBlKNFiI1+m54TTlYc1DnZrKdrUsYzOw8NUxD8r9bFsIJVskfVfbZHco9qslBzHfTMY0XnSFCS7XFD3NLMfkpV+cOSIqSli4hqjHoBaVxiVqoAKefVjbbAunl1dxuThrJObMcpampurPo/rltVWEilIMY/LZKL5TdKoXE+xaOjgJsV1bE22kT6sGyl8E8ECuFSDKnSPACTZOp335Hdxy+EiTwsq66c861V1Yybp+iJLXK3C+M7iR6szDJkkXpF6j+u4sKx0msYOnExApx6GC9KhdIa+O/lqqfpMzVU/bhb8gCdvNQtrMTMOs474XVZKD2f8soMoQmGMsp9FOM/1Y41FV7K9TKaSObAG6V6JYuIaoz6AWlc4GjqurOI3e5CXBUflugcwOZQJbWTC5S8OQa9tLgRB2uFZcjnpvPIvfKKHqFikwkPs4AXZRVSetBoL6dcgt5irFr/PaAhpunzKgCvASySlxiquYENJx8Hz8LBQvYHgnYpU9ll9HUuqz1hW4u7/TSn6eiS3otvf5ivCovJeOa2VmkaUYrj3c4+/Bel7naUTvaRppJwe2XynbkW3khQgq3R1GWmDWT6i85D3FUjb0TwydqRJuaOTX0Ib3YhloTz/hRZfdXPOu0EQNxAFYuIaox6AWldY68xZLMuoWawTgKzlyLp48r42DsO0Ay01TyNoWA9Zlovg1zb7SbYdvuopFqVF7M1yMMX2SQvbV9FcSMM+Dg2jfH1e0fYCrMt6S6kBNsEDu5FG1+lQSuyvWAbotYY0bdX4mzQzGUQ7+m1/SqeJurtHNpuqz8p0rdgMUTYsvmuCVwlLgYTVZ35/6zdzWyd6hQolYWiwLkIfaRopnyAdRZJuRXe1LSpSgM6yKcsPJCxDs0ufSPq1b7o+V11H/naZ5Hu/bBnpBUNubRbMBCgSQtgShZct0FdGN5+IiWuMegBqXeFoDHynwD2r+Fk8gx59ciee6qQr93tzkR7sHGAKrAchW1bLTWSBK5azPFmL60jfsY8Uw/JVtG5UIkn96vOZW8u/N/TZiKeuk+tI0VlaWuzu1lbIpkj20q3lWc1MtSZdN6SovJ4VL2jQUCHg/8YkRSolLDIXdtmJNDdA06tY56qGhpGNfFjv80wUjZhTRZmXUuVZqlhWncWU2CvLyEcMDCgdzc1Ekaa36vSnixTmz8rgzucNC3AYGyDTdYGXUEV4zyG7DcTENUY9ALWucLQG/vCw2Ik/AZ2GkFbyd+RNun58vXh+j0ZwljiQzGxHoWBEWzstF75JnGZYK9oT+WofieWUfK48FuvAiI+rfFtFQUhBiaY1otmw8AVbhGTGgepz2oEW2pQdsa7Iqm3wJHWe6glmWS8B9DJaaByGPes23Q1FfueiZ/6NE9mpeHnvZypFdPvt/NsaNVUBqXHh77csf1FNC+5OY7Vq53LkK6Wq9d1taDCk4WR0TM170DQykk6E2NfGfWV8W0s39hZpVZ9OextD2ACZc7BHtkBuCVk3KyauMeoBqHWFoznwRYFcIpIjM3Fb/yTrfycqL6GNrkznSR/IVyNdWd/biXR1QcpkDBlRVopWXwEnLS1E+bwtAFcsp+TjXl3OlcRVDPAqsxz+x0nFzYjo2YZFaLyI2GUNA8paw8a41eixLkPKypmFDGAfxfJEzKWl3TyKyB88lxvxbjDljPz4nYv8a3kbmkKBfwITanKLEEvQDZM1Mt1MzxwkgNFJhosFXS43tGCsbOh2J+PQHXOA1XpcnTpCzARmln1o8L1ZUIkH8CxtbbQpKx9s6iohZyqIiWuMegBqXeFoD3xWINcpzcGOB60LtCkofyitqNNVKV/HEppVifI2o0VLnV10cJLdXDyEFPUgy7RAXHst+/LKxKCvr9pvVmMK6zpDSNEqzPeM0teh0TLcYPiyCiJgN3TnaVtSjcQsTtkjyOe3FH0vGM6jZr+EhiWt5vbDG/EFTUL3lUzCSrStpIwXNb4DzbRpWT+z74uFcCzry9HF/TND4rba3tko0Gpc4PvZOYuTQOdyxgaP9fV6tbiG3Rfm6czBlL9TBOv809Ym714iGiuGpJ74e6bs3qZsfmTqCDET2Eg9/jcLKvEArpLJjARoVOZHXSdanPJ5vdjiGuMYhEZEhBpi//79aGpqwr59+9DY2FjLqqsolYDBQWD7dmDaNODoo2vxoa/M9n29hcjhHiwEACSTwDXXALe9fw3Q0WFMHwqYhSL+1jYL/QvW4H2rOqFt3Vr92040ow8X4wHMwSBmoIwk8xqJBFAus6/fkirh0Y99A+/84VLvxhSLwKxZAIw+O/10wGxOAiVcjy+jCyuRxp7qTwiA5nHZLchgX3Yl3nbjPPuzmFrCDAwiudN4MKX3nY1N3/kNxq19FG+7/yuezS09XMDg+HOrz/Wft61C4uJF3vfJAAEgaOjAatyHeZiJtVgL9TGyGRl8F1firzgL2zENzdiFXixFG7a6vrsFGdyJK/FFLPPVZsAYP+swCwAwF2twJ65CM3Yzv0sAtO5u4LbbbJ+XSsClU9bg7sMdAIAERsZwGRoAQkKiLQSgHXnch3m2zxsbgf372b+ZizVYiU5m//iF9f0ERob1Zz8LfPWr9u8mUMKLOB2t2Ga7bxmUAWxFKzRo3N+bn2gen1mvKdPXsjD7IpcDFi4EsGYN0N4u9dt+dOBc/Mr2vm9BBtu6V+L97x4GFvl71wBgJ1rQjCG5e9U0IJMBXnjBmHDNyWnbNuZ8KzMnhQECUEISE/EqPojfqM8XmgasXg3Mm+f602M3r8U/L1O4nrOPQoK5fu/duxcTJkwI7boxYpxwwglISo7V45K4OvH0Davw1i/5n3S70IsdOAXbMQ2DmAHSksb8gzWgTjv55IEADDek8Jeb+vH2tleQuOhCkIMglCvTr0mmVLFsGXDDDcD9a0p4/8LTcWqJszhzJr01FS4OABfQGqxGB2RJjBVlaNA04PHrVmP+qnnYupVDWDIZYOVKoKkJOO887wsXCsC55478e+1aYLb/DUkZGrYig9fjBQCoEJqt0vfbhV58A59xbTDasRoDmA/AvqCqkEIeTGIyF2uwGu3QILFoDwyMPFiMcBnWM9mMNlyD5ejFNZ59Ye0/3ibLCqPNcmOqDA1DaMYpGPK8rknmWcN69Wrg058GhiyXuTK9Bt/Z3QFNgbha300Alftgk/49SNs2E0OVfxM035sEfOQjwC9+4fk1sy8se1Lg5puNycEDVCkJ22fGu4ybbpK6hhMqz9EF601YJyfy2YchYRaKGMQMtfkinQbuvJNJWgFgzUAJ771QckOlVd54DgkOgv379+Oss85CsViEptViOxDjeMJJJ52EU0891Xts1drEW49HDUceKfo+UnIK0pu6gWZApxksdT/Od/lrWY+lbP8WaCjKZJPhFTOwVpzyUSylks8TTW8NHnldttyHSEvSX05HMjq/UGCLVyoW83h1XsUPWuY3Q0j5zhIVpK2d6KVxGFZ7Pi0tVZcNpza7KGe77PVlslepRPOb43YkYIXn0uDWMmUNa6uUXtV7pbtbKQ2rU0GBl51sLvLMPmV/P0NDSHkfWVfGuDi73oiPqyvY3EOQvwRDuor7bDTN+L0gjaHIT3Y5uvyNd6f/JsMPzIjGF4+TsIuZ9EJaVu2ii4TR/6ZKiHQAIyNuICzs3buXbrzxRnrmmWfo0KFDdPjw4bjEJXB59dVXadeuXfT000/T3//+d89xiEhGtwD1SFxHfPrkJydeqlRnzvdcznsC8+NLpZrK0ixWZQXWYrk12Ub6Pf3CTCxB/dmspQvLxX3vJ6cjYwELEtyyALkq2fjTW9ulfrMXjYH0ZYMQWD8+smbfyboLZrNEz57fJd1/Xl9T8S21kkRxsKDmDuSRgRnF6fF8SjB8M3m6o6zsZKJbc35/NiR1UT3aalUe4e5JOZGrSkGXpmi2sz2VjbhTzks1yx1vzNonJ72qo+reaNUmG5p1bhbGFbS0GM7WHihYhgFvg3MDltHfPt5jaNQWCpFlz9q9ezc98MADtGfPnkiuH+P4hkledY/xixq1p4p6JK65HFU0IdnBNDylAS+r6E/u1j1JsV9C5SeVJas4F8t2DNDhFnEmllLfKERemzkdWX+zahZySEcQ4nojlnmLfjPGAEuOKYokDu7Nk4/7rFivZAO0czmSZrmd6PUUll8IhhQCo9yM613XmAd3NPrBVBs9cHme+voks2CaZte+PqmNUjmToU3ZPHV1ub/e1mZPsGD9XEUxKkyFgx5kCTDq54JjsZS2iOZy7OjXtjbSB/LUkmJb731p5wo0SnnDkkkik0nuu6YaoMU7DUtAp481Fql0d58hXCw9KA048yWw5uxaZc8aGhqihx56iA4cOBD6tWPEePXVV+npp5+mw4cPC7+HGrWninokria5DFuj818nFumuT6hlk5Etfi2uomJaJbwysagmCQildHVxrTnVtnkcefotKhsaa2EtZLLWpc3v76ht/ypYXBOoJIqokLyy4HjY6UqzA2m6Ectc5KUTvVLt7ESvrR0dzUV6vMuRLk81X7ufnNGFQvXnVs5r5SXDw+6DC1lXDJWxIlPMja7zdN3lJjFsPFtre2TboReKnIsagvyinytpLntkhRJtvqz9Xegpkn7vgMBi77YSVzN7MC3T9o2qta6PNRZJHxaPSZbLiq4TfeITXv0mcLMKmbwODQ3Rf//3f9PBgwdDvW6MGEREhw8fjomrLMI8+raW5ehS1uT0KmUY2q1+fFybmviT+GwUvMl7xcqxqs+fZFOgUixyrTnVyTkCWRyvZyHzPesm47SMIZXGT1Wq0Z6GNtp4ezQbHtFzJRohVjzj9jywZcpUTyXMYupyfhx3S7X147ibAGPB3tvoaEdzMyeHrQASbgHM0tNjq4M1NHlGL1OKi2UBNPsjkQgv+5t1DFpP13ltdrqUeyceMTZo01t1rg+x175gnCapuZxOe5Ix6QQcBYOkL0cX7YDdbO70R36qx8ImOZZpLx/nwy18KyjrWaTTRuFtbjwt1RFkz4qJa4woERNXL1i2t6Xr5XJXq5Yg2W5E13QSV9k83/fcM0JK/Ii9mytfsUj+AypUi3PyZUbSVBCBEHkYZdP1OXtzhf6EWnXB3JbkE9xQ+9exmJrNc2YxMlNOhnkyYd7zo5gt9f1O9NKV6bx3v8gclQa10Ffq4HFfntGrWORbykzf2R5kaQFy1IMsxyIo279gBmWJ2sy6FN+X2Cim/yzvfr2aKm1dtli6vR6rprHnRk0jYww5nv0ONHN9ll06/pZ5qNBj/z7v2Zo6tU+2Z+mpnhzpBcMKK3IfEW1upPssRC3XmLiODorFIgGgV155JdJ6ANB9990XaR0ixMRVBD9HgwrFMwo3hGJaUEQTm/UnF11kTL7ZbMAc2n19pOtEHc3FyO7NVWStaDW2uMqWTRf00MZex1GhpNWmVBFct11TxULoNLNbiyBiiZUAQgffHzBIUUnL+9Tn7hbnpbf2EYNFWfc9gV1eNI3KmkZXpN1BeIBBmmbBCBR64nYjDWixSJT74TDtQIvQt9L67yGkXUfWL0MuYLEM0DxHUJYMX08m3cOsHQO0F42uLw8hTe3op5ko0kLkaH6Lfayz9pNOQilrcbcmRfGa4udx5sb/RDdz48PzSwfE3M867XhZQZ3Pdqs2MlezfFdZG0VlNYYQs2eNdeL6m9/8hhKJBH30ox8N/dovvPACAaCNGzeGfm0RcR0eHqZ0Ok233HIL87c333wzTZ06lY4cOeJZT0xcOag5cXVa6Ew9KMaE4mdRZk8qARZDybIAOYHVZmQCbmwcccsyJ8etiQBSVr29RESU7w/vGJPbt8mk/TMvK5rXOXdU7ZT8fFsyQxu687b2FpeJLeVzWUfzbW1EV18t17677x4Z/7J+oBVzXBQkNWj5w2W98t93WOude4UwTg2ssm7O58YiTbeim3Zqam425jtuWmFnokjjMOx5tK4jQe0YcO1PVPZ3ptWyB1najwZO+9xjfXhKytgl67qrPlbf7GEQYmapzD+eyBtWeZZFW0Y2zCqh5nXabp12VP2Szbn6VnS7+kRHQhgALLt5qUeLq+jgLEp86lOfos7OTmpoaKBt27aFeu3RIq5ERJ2dnfTGN77R9Xm5XKY3vOENdN1110nVExNXDmpKXFmWVScZsi1C6uR1uNEt8SK9IKZSRA3sxcCrmD6p/J29RkMT3Qtq4IAPi8VjQ3d4EjNlgHbhZOrC7fQ1dLLdLGQCDjjH8FEUcxGU9e80+8okr7pOtvSsvJKATt/9eLF6vFid9WXaqbpoRRTgFsb4eAlt9JmUnPqAsw+cR+MqOrQyxRrIZKQvZaf29btBthIqk0iKrmX+rV3L0z332B+xjEdNAjp9o71IdzZ0Ma2srPqYf0unSR/I21yUeC4SUn0hY3ENYQzPRFEpvskcXwt9KEGY48LPeCw3S6qthIQwiKuKP3iYOHDgADU0NNBf/vIXuuiii+jLX/6y6zsPPPAAvec976EJEyZQOp2mCy64oPo3Fqlramqiu+66q/p3a5k5cyYREc2cOZM6Ozttv5szZw5deuml1X/ffffd9I//+I/U0NBAp5xyCi1cuJB27NhR/bsXcf3jH/9IAGhwcND2ufm7P//5z/S73/2OzjvvPEqn09TY2Ej//M//TP/zP/9j+771Hll1bty4kQDQCy+8UP1scHCQ/umf/olOPPFEymQy9JnPfMY2Pr75zW/SmWeeSRMmTKCpU6dSe3s78x6IYuLqP+hCtlTeNH3YiGz2E4VLt9+uPlFVyo24Ser7WfTYrHmBJXYcREgfyLvlswKWo6IjaZnJWNUVxBwnjZJWH3iTEJGlZGuyjfRhtyVKYdhVF2dR/S+hjfL9iotWnbpblAHfmp+lvpwrkj9oAg1nWY4uKZ/xoFbsHmRpM1olx+gI2VWxuPr2f+cVzdisJcPod+dGjGW6C2EML0BOWcc/n6+xCxUgp7YSIoISV1V/8DDx/e9/n97znvcQEdHPfvYzOuOMM6hcLlf//uCDD1IymaQbb7yRnn76afrDH/5AX/nKV6p/9yKuv/vd7wgAFQoF2r59O+3evZuI5Ijr97//fXrooYfo+eefpw0bNtAHPvAB+td//dfq32V8XN/73vfS5ZdfbvvskksuobPPPpuIiH75y1/Sj3/8Y/rzn/9MTz/9NH3qU5+iU045hfbv38+8Rxni+te//pUmT55Mvb299Oyzz9L69evp3e9+N1122WVERPT73/+ekskk5XI5evHFF+l///d/aeXKldx7OL6J6/CwvGg9o9yLDuYOuLroVI7ATDilXjyjgU3y1efDelRp15CiWoGyQz+r8AijogZmKMXLkmhd0LJZIZE93NJGj3XladOyfmlicbgx2H1u7C16JgTjBd1V1b+uE2sP/ye6q49M+mjOZ4CbX8kw2WLqkPqJtN+4vOBP2F/x/mvhIuSnnk700uLUiLVeZJCUzvakWtraaN2ygP3unH8YG9TDLRn680e6Arf33yYX6dVXFdacypRTeFinvVPCl1bkFhm1lRARhLh6GcIjMBDbcPbZZ9OKFSuIiOjo0aPU3NxMRcs68oEPfIA+/vGPc3/vRVx5rgIyxNWJ3//+9wSgqpcrQ1y//e1vU0NDQ/U3+/fvp0mTJtH3vvc95vdLpRJNmTKFfvaznzHvUYa4fupTn6KrrrrKdt3BwUFKJBJ0+PBhyufz1NjYaCPHIhy/xDWflzt/FZSZKDKtDgfT/Mkgn7f7knIztVi3ljW0blnTZKpKWRnWRYUzsxrcz1M9ObUJzsrcKr6ej3flqKPZLi9z34kXCY/+N3d0GQTA56bDLIOfzgmHqSjoTtMMWa3DzeLUsaa1jcXbuUdzPsakGVnuPFLei4bAC7hxHxmpqG3muJ3SSK9Ntm/y/EjUCf0iE9EGYsq2Q6pkDD/rdNr9pygs0bZ+klRvYUXjk6YZOmLWzSjDdBcG6R6qqLac3Gj4oMs4YprccS7ybu3XqEo6Lae2EiKCENeovJtk8Je//IXGjRtnO35fvHgxXXzxxdV/T5w4kX7wgx9wrxElcX3iiSfo/PPPp7a2NmpoaKBJkyYRAHrqqaeISI647tu3jyZOnEjf//73iYjou9/9ro3Ivvzyy3TFFVfQmWeeSY2NjTR58mTSNI2++c1vMu9Rhri+5z3vofHjx9PkyZOrxWz7008/Tfv376e3v/3t1NzcTBdffDH19fXRoUOHuPcQCXG944476O1vfztNmTKFpkyZQu9///vpoYceUrlEtMQ1BPeAo0hSOwaqE7nVUlN4WDwZ6LoRvW82gXnkZt0JBwwmKqfUFmDz6HA+7lGyhpnWO6WUmYptUy0zUeSSL885XNdpXdaIgBbliuc+NyJaly0Gav/A4qLtI+tY4/kvmp+pWM55iSq4R3MyY1LgJ+6UFGrHAA0l/FmnTULcjn7Xn9sx4NLedP6W767h75kxE3NEMLbDbLP7HviR82EmPGCVh068QOp7zue6JdFG+nWMVGSCvjJUXfw9nx5k2fNBJmMnz5bkCiL/3ciKMA1aNAhCXJWy8oWM7u5uAkDJZLJaEokETZw4kfbu3UtERKlUSkhcNU2jNWvW2D6bNGmSJ3GdPXs2XX311bbPPvrRj1aJ68GDBymdTtOiRYvoscceoz//+c/08MMP264lK4f1iU98gv7pn/6JiAwL8yc/+cnq3z784Q/Te97zHvr5z39Of/rTn+i5556j5uZm6rUEPFqJ67p16wiALb2v6Q5hEtc3v/nN9JnPfIaee+45VxkeHiYiw7r96KOPUnd3N73hDW+gM888k3sfkRDXBx54gH7+85/Ts88+S8888wxdf/31dMIJJ9Cf/vQn6WtERlxDCioxFkuNKVHS0VwU+gyyeLP5+4XIGWTHyaIGBvy3V8En01pmolhNcSvbJy+hjU7L6LbmCwmiNcF2iMXqt8ciX56O/3m3fqMZnS3rGqLrRNNbxUfWXj6uP7nbbkGUTSlr6vgukkyRKkoNzD2a4wW4mZ/193Nzl1p1SJOVZ5TvNwbK4105un3C9crPfDMyNM9D3N2ZoctrYxaGG8MrjW20/v92Kf3Gn3KJP4UHr5TUzsDNMFPMBmnXOAy7ZKH86BmXK9dU+f5OpKsyVDIEtJzJ0BXpfGBrtfLztVpba4ixaHE9evQonXLKKbR8+XLatGmTrZxxxhn0rW99i4iIZs2aJXQVmDp1qs06+eyzzxKAKnHdtm0bAaAnnnjC9rsLL7yQ5s+fX/23rus0ffr0KnF94oknCABt3ry5+p0f//jH5Ie4rl27lgDQz372MwJA69evr/6toaGB7r777uq/N2/eTAC4xPXpp58mq9WXiOjOO+8kK3FdtGgRnXvuucI2WXHw4EEaN24c5TlWsJq5Cpx88slcHwoWIiOuIR67mxMoKwf0LqToqYuyrklDhjc3NY2oE1WF6FlndgBRgi+FEnTBNcmMl9XKWWaiWJ1UPAliBMkAWDqLVvLl5fi/oZst8yQkMQx2Zw41kSg7O6q8oipwbX81reaIwLxaX3RBLrBPJjUwc6EQ+c1JbhJd8l9E9MTt6u+p2W/rr83TFWmxcP/XcDXtQIt0f/p5l76BJeqBmIJx4dWGwy0Z7kYhSHGOjagsrqz3jdU3LEuwSQj9PCfZgDlruQHLlAioacHuQbDnwyLZwnkp6hB8DsLwca2hCAIREd133300fvz4qmXVis9+9rPVgK1isUiJRKIanPXHP/6Rbr311up3FyxYQG95y1vof//3f+n3v/89nXPOOXTCCSdUievRo0dp4sSJ9KUvfYlefvnlan3f/va3adKkSfTggw/Sn//8Z7ryyiupsbGxSlx37txJ48ePp+7ubnr++efp/vvvpze+8Y3kh7iWy2U688wz6eSTT6Y3v/nNtr+9+93vpg996EP09NNP029/+1uaMWMGTZw4kUtcjxw5Qm1tbTR//nx69tln6cEHH6Q3velNZCWuTz75JE2cOJEWL15MGzdupGeffZZ++tOf0uLFi4nICIJbuXIlbdy4kV588UW64447KJFIcI2dkRNXXddp1apVNH78eBsjd+K1116jffv2VcuWLVsoEuIaAVESBUKUHakHVXnzFem81GQsM+mrljlNxeo/E9ApCznfswUwMkBJRYZG4L/7EiOxglkKBcMKyssglqxmovJZv4XdWYcay/r3EtqYuoxbk230lznuI08/bVrZcD2V03wdT5XUwNyjOY5JXTZFctkcFJb3RH91mIYSfGLJvR9NI8pk6FBK7NcrrW0ZoO+thE8lWIw3LrzKX//5UiPgNGSZMqc1foQkhucCoeKOdBuWut7fIGTaeo2vY4nUb2TnQvu405SDZUd+C4uRxK4WsRNpeq3BYdiIKOhKFmGpCtRIBIGIiM4//3xuwoHHH3+cANCTTz5ZaV+e3vWud9H48eOpubmZ5s2bV/3utm3b6F/+5V9o8uTJdNZZZ9FDDz1k83ElMvxK29raKJFIVOWwjhw5Qv/xH/9BqVSKpk6dSrfccovLxzWXy9Hpp59OEyZMoA984AP0wAMPkB/iSkT0la98hQDQbbfdZvv8f//3f+k973kPnXjiiXTWWWfRwMAAnXbaaVziSkT061//mt7+9rfTiSeeSDNmzKCBgQFyymH97ne/ow996EPU0NBAkydPpne84x1VqbHBwUGaOXMmnXzyyTRx4kR6xzveQffeey+37ZER1z/+8Y80efJkSiaT1NTURD//+c+F31+2bBk59c3q3eIqPSFb3jQV3mwsEK3ek3oi4fps+ER/7gFGmzU63GI/pgbkLS0zUaRCYWT9ZEW9J6HT/JYile4OX2FgNgrcP+c6xBnEglqTSn057lAT5RL/4WVFWr8kZ2TOuped/MJPeXre9XxrfWV8HpooR1xVjubyeaLFKcVNotUkXmcasUZQ2RTagPdKfX+IsRnwSoe6HF30yTcUq0fgC9FHneil/8J/yLc1neZqPvvVh3UScKsGba2TeJhj1vrvzcj4ShLBcoWQff/9EFf/Y2/kBIkXjLlpmeFTW+oz5hAz+9ooeAkQUXQ6rqPMx2PUCSIjrsPDw/Tcc8/RE088QZ/73Oeoubm5Piyuo5Q1yVyUVXizXxJlLoR+F4UyQJTPM4mXyGpkLgSnZfSq6ypromWlpgxTEonnrzkXvCw5IwtDUP+9juaiUkxdMmm4L7vGZ1jjTlIDuKO5GNrRnGkt8TV+OVHgY63cgGXMTQrP8m5unAJli/IoogxLrOIkdsxAJMam+VWcKJWMIKzix4WGl7ZVbo7LSLvgsMpeTFH6vjk+REkZygBtuLZ/VAT7WRjrmbNi1Ddq5uN67rnnunS8RIhcVQD8TEaRTbLFoovM8KxwQLAgiCDEdSfSdPMyvXrq6M4kxLIajSwE99xD1NPDj54Nk6SyCstf08sFwNS89bsgmYt8NdAoPzLURAm6+vsdYzOkE4EyYFjfJOW4Hu/KhXI0Z+XdCejKkmqiDHFlgI6eODnUseI3mEnkerEXDa77tlr1ee984GxRwvGpfo8laNVgN5VIePN796Ij1GfldX8qCgHWzYKziOc4BJaxuhHLhGPIdGVZhL7q+JAJ6rIq3ZilFoL9LIRFXGPEYKFmxHX27NlCIV0nIiWuuk6vNYjzd0dSKk6CVlmUKI+tzUnQz+9mokjNzUTnnz8yAVondpHVKJkkGodh2oHmmvZx2UIenZP3LIW+NBZAXh38FK3t6K8qQ8xvKZI+bJgHBgbc3hC2Iy+rWaEnxCPIfF5ejquS7tSZOWp+ixHlL2vucPJuP0e4noVh5WM9J7kx43es8ceGV1ARi7hGrY3q5/5uRTc1TtZpNgq0Cylla62sH3GYcwRTIcCcvLJZWt3hNhCwCmuO2+mlKuJVNI0OpdtoeqvOVWvhjZPvvKVX+v6dhDxqwX4WYuIaI0pEQlw/97nP0bp16+iFF16gP/7xj/S5z32ONE2jRx55RPoaURJX2aCR0IvFSXBDN8+6otkmLikfV9WJXKI4j9udkpwiS/Fc5Gt6VFidnSspI1l+UY93yVuv+VHdhtLAq5PsFhdeMM3wFENZYnqrfZFsbrZYWqPy5+zulpLjKkGjcmZkVTM59GNdjBS9Zvpi3vGdrlOhJ7ygGdGYrunYUmiXlyVtJ9LMzWrQaPOwC6+tqkVGuaHaXyG4iLAUAsqW83JdNw4iRPOXWazfmY1CQBkrjcqaRvpAngoFY3+amzdAByfZyb3VLcBP31sTijj/HIVgPw8xcY0RJSIhrp/85CfptNNOo/Hjx1NLSwude+65SqSVKFriuun62ugQVotzy+uRP97qW2YcW/lfrHuQ9XW0JSOPxCpB2+u7WEyYTGKleARvHj266ujvp8L1Bcqih7LoodkoeOo5DiFN7ein2TB+dzN66BwUaMO14QVhsfqjWPA6gjY2SZuyjnNEjhyEQdw1uqzRbtHJZCoSYhk3IWtHv3LqVZlSBqQsryplx8wO3xs95ba7nkN09YXZVtWyHF3y9xZCkGYPsq6o+22J1qrcmq4TXdLgJoV7MYVuxDKuJVZ6A9bbSzQwQIfS7hOpSxrcmcimtxpZt57qyVGhx0he8+SyvG9NXrN0otd1L7lcZW4sGBvMp3pG0vuGjZi4xogSx2XK1+9dXKzJpG0rXV0jLEqSRJnkcR7ybrkTyfbfhusMq5rkvanIIzlLGBZi5fKRjxj9OTws9uLXdaEsFK88t7h35JoDAwxy1kpDSAsXZ54VTjVYRrUUeozxw0svuRNpmou8XebKIziMFYk9r0KMeS4Ut6JbiZyNlkV10wU9TE3mWpTIiaumCf2Hoyo3VPw5pb7f10d6oUiLUzm6EcvcG0eP/nsFjUJXng3dedqUFfvq7kMD0/dVOt4gl6N83vCp97Lomo/F5oPq8f6pvBtWtzPAiHu8Iu0m7YfS4UdwxcQ1RpQ4LonrDdd7HZ+GN3G7tA4zGYPESvx2AXJVQ2K+X6dzKha71bhAsm6xvybvN36Ja2ApKYB0RWuXXijysxyYaRf7+uiF2Zf7CgqryltxrJD1TMae6skJA35KMPzhbEeIkpsqU3LMyzfTnqSj1fO65jMaLQukaSU2/Tpr3Y5I6jPZUQTJCfj3YUTfKwXnVQai+aoZmbC8fUpl3usSNNqayNBBgb6v9VrOQDpZ+asjjxSVPX9sB3IhJ8gx3c7S6ZENJuvkpYxwI7hi4hojShyXxLVQ8I4cjWxSVzga3thrqBCYvormDr4TvdG1r1JkXAWcfmKy6UXNBUL07104mfaiUbi5eAlttGlZhMftgDFYwpaoUik9PfTMv3X5OsI+8t8F2poQC/Fv0TLVIDIikhYa3oUUzUVeSds3AZ16kPUkGsYGRtINoKUl1Ofv1MxkzRFRbkJYzznwfNTSYmyWH36YKJkMPRjKbeFERVWgXf46KXtqUnMvKuvrGfY9vYQM0/rupQRwamrYd9XFovz7J1vMjeMpJw97bjCtvu5BERPXGFHiuCSupoM+Lzq+JoESyaR0TrtNWXc7VS2pqoWVLcckqT+8rEhXnOSe1FWsK87gLZ7fH3shH4niP5iKmFBmokmjKVvuurRIySR7rPLdPTTa29hGFzYXpOr400XZkZdD0uJjRj/LqgZYx5Ps+7Ufk8WEJJkkuu46sd6Yj2JaC2ejQMvRpZTuOCiBKsMtt7Q1kTEy8Knex/s+QMNN4Sb3cJadDE1mlSN+s/zt8ixzni4WiQY7B+jI5KZI74P3LFSl/JzH8yollyN5FRDFIm3sCCmCKyauMaLEcUdczcnQPK0fdWkaL+HMPE80v3ZaqDzSxLK0eE3s5t/n4x5PmZ0SNGZksxl5G1XOdOHzqVFh+ZJax2oOF3nK6cj65ZUB0gcUsiZY6pKVPLKOp1AlsjSNqNudHjfsIutmEvSdtJLmBcjRLBQp369z5wGvtgROROJRNqO1KgNnZtNSrfMVNNKqPo6lj5c3ugZFFAcgen6sxAYyxUxHHUUw43/jX+S+y83rrIaYuIpx6aWX0pw5c6r/njlzJnV2dga6ZhjXGCs4roirivKQiuC279LVxfbNzGaNCaSSN1V0PBUFeX0ZLZ6i6F5t8joKfgltNBtyFsHZKDADHa5sqI06RK39U70WP9HYNPU3AXmf4xJAh1ssx4QVsiB734bkEXuhLcOdfSnUd8o8nRgepnVZQ0M3CleaWo+BmSjatH7zecNHUSVwLAzrr5dLj+kWMKIeoX79dgywDX0huuiUoNEuLeWOOYhgTDg3H166seYQtmYbZLmo1KQc5xbXSy+9lMx08yeccAKdccYZlM1m6ejRo6HXYyWuu3fvpv3790v9tlgsEgB65ZVXbJ+rXGOs47ghrn427u3o93XspTRJWLWbsllfE7WfoCPRdX757muopcU78CZokQ14yKKHuQB8dFKx9hO7z35VeXairD7ewVCGf56fkwO9UKwOxU3ZPJVOOlnqd19DJ9sXVLMT8CjHk14o2jJ21ZOYv+xYsJanenJO9bzqvXmfVNT+nl7xodtcBug/0c0Xxw8pUMkk12HNk36KlwtBNmtPcsdTAolq/NWtj2uNc75eeuml9JGPfIS2b99OL774It1xxx2kaRp95StfcX13eHg4UD1W4qoCHnE9nnBcEFfext1LhDqqo2jzGDjfb3kJAxyJ9WNeuG1sbiZ9WKeNvdHcv1luxvWBFoB6JijO4mWBdqZ45H1ddkz2IEuAmoVzccru17xUMv3tf+E/6EYso11I2T4/3GIn4FG6djzVY2976JZdH4UdvS35e4vli8XfxJmXwr2PH+Li0PvmZbRQB/qrXlHmfLN+Sc4ISh3WQwtU2oJW2uUhWRf9WDA2cZc08Mlro4X7m3NbFESbJxfm0nMOgFCIK08tJsL8tSxC+aEPfYje//73V//2pS99iaZNm0ann346ERFt3ryZ5s+fT01NTXTyySfTxz72MXrhhReqv9d1nZYuXUpNTU2USqWou7ubLrnkEqGrwGuvvUaf/exnKZPJ0Pjx4+mMM86g733ve/TCCy+QaRE2i5mR1HmNPXv20Cc+8Qk66aSTaOLEifSRj3yEnn322erf77rrLmpqaqJf/OIX9OY3v5kmT55MH/7wh+nvf/979TvFYpHe+9730qRJk6ipqYnOPvtsevHFFwP3c1DIEtcExjAGB4GtW+2fzcUavIjTsRazsQqLsBaz8SJOx1ysAQAkUMKH8HDobSlDAwB0YQW6rk2iVAJQKgGdncar6QPN2BViCwHs2oXkV76Id63/ZrjXdeDfcSd2IV3tEyfMN9OKVmzDanRgLtagjCQ6sTLSNoYF9h0aSAA4BUPYhlaswyyUkeR+dxq2S9V3M5ahHQO4D/OwDFmp3/xpzzTbv3fgVKnfLca3kEUWaewBAOxGCk9flMVPe1/AfZhX/V4rtkldzw+2w952lfuOCrvQbPv30YaUcBwAADQNaGsDZsyofrTd8cjnYg26cbvr3QCMcRb2ZP0STg/tWjejB7NQxOuwHY+3zcfq1cDrfrsGOyadjnctnY2z/2sR3rV0NnZMOh1P3/+c73rMueO+ZAeWYynS2B1av/iZpRMgaCB858ROXNheYn5n//6R/56JtWjDVu/x4gPOa25FBh1YjU1nzWN+f1SwZg3Q0eFeuLdtMz5fs6ZmTZk4cSKOHDkCAPjlL3+JZ555Bo8++igefPBBHD16FB/+8IcxZcoUDA4OYv369WhoaMBHPvKR6m+WL1+OH/7wh/jBD36AX//619izZw/uu+8+YZ2XXHIJVq1aha9//ev485//jO985ztoaGhAW1sb8vk8AOCZZ57B9u3bsXIlew287LLL8MQTT+CBBx7Ahg0bQET46Ec/iqNHj1a/8+qrr+L222/Hj3/8Yzz22GPYvHkzrrvuOgCAruu44IILMHPmTPzxj3/Ehg0bcNVVV0HTohiVEaE2PHoEYVpcnRt3r0xCt+HayFKWOo+Bi0XyfSRm+lLtxZTId+VRFOsRHktySCazGGC4dKjqv9ZjcSo5sIqK1fIoktSOAUpApyGPIDhWmki/FlLzPXpy2cg4n4u8mqanSkmnqfAwO2XnaFjkR4IKHbq1zrzJjmKMeY0e68pXT0V13UjGJHtPYb635riQ9UU3fsNpl6bR4ZY2uuF6nXp6RlTmxKmvQa81qCcNIYBKiWhcvMJwNzBPQ3gnfnORd51eRFG+jiW2esNMCRvI4url2+zMRBkirBbXcrlMjz76KE2YMIGuu+46uvTSS+mUU06xuQj8+Mc/pje96U1ULpernw0PD9PEiRPp4YcfJiKiadOm0W233Vb9+9GjRymTyXAtrs888wwBoEcffZTZRp6rgPUazz77LAGg9evXV/++a9cumjhxIvVX8o3fddddBID++te/Vr/zzW9+k0455RQiMnxmAdDatWsleq62OC5cBay8UMZHMArSdggTqQdZZho+P0diJjmISrqrVr5gJYD2osGXL7E1Uv1G3FR39xbkfnhFlYyVAVqDOcIIaavgurWMwzDtQLM/4gCDqExv1bnC52GVMkBXpLwC2WqzsTGj6v0SHJPUAIZkn1MFqyZKGrAHCCagSxMp9ibUHWyYyRD136N76AxrNKTJH/GXYGS+ijKBRVixBLei2xVotxkZ5SxzQYp1vslkwuWBgYirrCEnTKZdwaWXXkrJZJImT55M48ePp3HjxtEll1xCBw8epEsvvZTOO+882/evu+666vetRdM0uuOOO2jv3r0EgNatW2f73QUXXMAlrvfeey8lk0k6cuQIs40yxPX++++ncePGke54qO9617somzWk5+666y6aNGmS7e9r1qwhTdOq/77ssstowoQJdP7559OKFStsbgSjiePCVWDGDCCTMU7hZmAQbdjKPTZKQHys6xcn4jCyuAlzcL/t82nTzP9Rg3m88zzOUP4tVYoItToMSABowkEkwT4+E8F6bP4l9GAX0p73tQ8NALzvXwZlALuQwgDkjth4dZYBbEYbBmEcD4tOYlTdIwjABY4x58QupHE/5tg+m4s1+BvOwFTs8jUWEiCcOLQFd39qLVagEwCFfoRtggDcsKcLCZSQQAmz8UtkcQOyuAF70YQL0Y9taI2odjcOogEEf+/QX3FW9b937zaKFbKuIkFhzi/3YR7KSFaeoTduRNbV11ss1zKxbRvwzQWDaC2L5mJCM+3Gq5goVbcG4AgmgMB2lwjjnQ9jXiQAn8VX0Qr7MXgrtuKz+Cq0CN8Vs37rfAMAV14JJPkeSrWF0zcm6PcUMXv2bPzhD3/Ac889h8OHD+NHP/oRJk+eDADV/zdx8OBB/OM//iP+8Ic/2Mqzzz6LRYsW+ap/4kS58R4GTjjhBNu/NU0D0cibctddd2HDhg04++yzce+99+KNb3wjfvvb39asfUExpolrMgmYbiC1mvidMDtwBYwF1ubKNmMGkE5zf1uGhs3I4BwUsBA5zEIRr4fhP/iGyUNK7RjAPNyO6/zfiAUH0IByKFfytyBY/RrLSOIq3AmCJmzTFByE5rM+K8qVq1yF7+Kb+IzUbzS4F0+T4KzCApSRREMDkErZv5NOA42NI/9W8d80N2K8+9UAtGA3ZmCw+tlcrMFqdLgWVj/4Q+9a4UYxDCQATMcWXI8vYwdOwa9wHm7El3AjvoRf4TzciX/H93A59mGKNHnxS3IIQCMO+r5fp6+u6t+DwvRBNecXE1/BF4QbQ2OOasNX8AWcjhcxC0UsRA4XNI1cK4ESZmItFmAV/pnWSvs8T8Zhz+/oSOIm3IRmgS+r6J0XbeYJwGFM8GyD13VMmO+ks51e76pM3bLowgqbL/0Z6vaP6CBryPFh8JHB5MmTceaZZ2L69OkYN26c8Lv/5//8Hzz33HOYOnUqzjzzTFtpampCU1MTpk2bhscff7z6G13X8T//8z/ca7797W9HuVzGunXrmH8fP348AKBU4ht73vKWt0DXdVu9u3fvxjPPPIO3vvWtwnty4t3vfjc+//nP4ze/+Q3e9ra3IZfLKf1+NDGmiSsAzJsHrF4NvNoY7cQvQgKE6diCGRgEEdDebgSOlfL3uU0rFRjEhnANvsYM2nnhUItSG76Jz+Cz+Cpuwk0+7sCO29ANeBDFqKAj6QpKuw/z0IHVQutaWAN5N1JVK9IgZmA3Ut4/gntRMsnsQtyDBEo4dMgYCtkskMsZ/79njz1wAzCIxBZkQuv7duQxE2sxDkewMkQL6b4Dct8Lwxp2M5YhDfd7dDL2IIsvogkHpEmB+b0SNOTGXyrdBr99ZhI/qxWMhUHMqDx3f/TGi3hmcRMe0+wBggmUMAODuBufqHzX/VtghAyVkcQ6zMI9WIj79xnXYgXD9qJLqs2iOy1XygKswrN4o9T1WNgFw3DA2lgSgE1vWyh1ndEMW5Gt+y5cbtuUAMDSpTWNdxLDekTKAiOAcbTw8Y9/HM3NzZgzZw4GBwfxwgsvYO3atbj66quxtRJY1tnZiVtvvRU//elP8Ze//AWf/vSnsXfvXu41Tz/9dFx66aX45Cc/iZ/+9KfVa/b39wMATjvtNGiahgcffBBDQ0M4ePCg6xpnnXUW5syZgyuvvBK//vWv8eSTT+Liiy9Ga2sr5syZ4/o+Cy+88AI+//nPY8OGDXjppZfwyCOP4LnnnsNb3vIW9Y4aLdTGc2EEUWXO+snd0WQmUSmLtJEgHCOwiO/fWYbhA8ryh5qLvHTgRAmgIaRoNgqBs4MZubzbKAldOpd42MWaIcr82Ax28COzpVq3qZVqfhyGr7Hpc2bR1BfGKPAkkYKUsAKoVAN7RlNjU6ZtuxPRySnxEk44g3fGYdiRoUp9DmNnvLPX39ZG1N/PT4vt9EcX6Q4DomDYYNm9CEbyC7NuWf/fTvTSQvRRJ3ptEnSGdqrbl3cIaboQ91AJibodo7LP/igSNA7Drj87EzYGRWA5LFMe0iuzZMgQ6avy/rZ9+3a65JJLqLm5mSZMmEBveMMb6Morr6xyl6NHj1JnZyc1NjbSSSedRNdcc42nHNbhw4dp6dKlNG3aNBo/fjydeeaZ9IMf/KD695tvvplOPfVU0jTNUw6rqamJJk6cSB/+8IeZclhW3HfffWTSvZdffpkuuOCCahtOO+00uvHGG6lUKnl3ZMQ4LoKzrCgWRzkzCUYIiorWJHuxAXOi9frtZrTSlyZk6Ws+UzQSQH/+WDe1VLJ9mgvsItxNe9BUsyhuazT8aBBoa3CDEbnvLwLaLE5VATOanBV9HFVmtzCu5wzsEW0US/CX277W5bWGdGRBMyzixxrPOhK2fzv7zS/5N+tferWhpVrqM8TeN1w7wCWcJYCWo4urO6xpJJXEhKcqIltuxvXV92Mh+oRZ3HjqGdY+Z2ntmuP58RM+qNS2eiS5m/AWyqKnasBwfiWsYP3IdFyt6eRiHLc47oirqbTBSp94FMlQFiYZ2aEwpHr85k/3+rfXpGsuNDZrZ2VNDdMKKNs3PcjWLBLXWpxE07x3mf5lFaeqwJIlbAKzGa00NMqC6qLiJGK8jeJoZzNSLX+6cBlt1VpDu55JAD8xsd/2J9lNidl3JoG8DddJ96UzG9085Glb0kESkkllCTVgxCjW3y+f3c6v/OC96HC9H6zNuFcaZRmCvQ+TldrGk/mr5ZgVlSGkmf0RRrD+WM2cFWNs4LgjrkQjpxBJhyWrHWwLg+pk5XUcl4AeST51XnvC+I57IncvXDyLRVTtNssupEaFxDktrn43Ik43DvNPD1zOP2Kt9b2KSheWe+Zl5x0518NCLt2GVIqOrOqnWyZlQ5NcMtxORt4jP3JnO5Gu/l5WFs46doNY7z/WWKR02n4qML+lSPl+nfJ5ogWQk/o7iEm+xgJ7rlVLowyEKzNmPtN2DER+ChRk48eTwuvrC77GhkZcY8Rg4LgkrkTsUwhzEmft4FUmC9bxpzlxjpZPaFSlE71VC3LQo3JRn9dTKQO0S0vbCNqcpmIo92n6Lp+W0anUGswibxD66N1hnJZnnrC69fNabdz8PAOv8tRFhpvNDrSE1gaTSPolULNRqPax2HKoBSLKruv15UgfyNPhFsecmcnQFek8nYNHRuWZmmmUP467Xb6srJ/IEmyv8WImkehAf9V94RtYPCp9IHsvTn/93t7g62tMXGNECVniKtaEGEsolYDBQcwb3o70FdNw0X/NwNAuQ991GrZjO6bhDXgen8Ot6MJKpLHHFa25D404CfuZlweMyOIESuhCL3bgFGzHNAxiBubgfqxGBxBKDHV9YAWW4gZ8EX/AO9HMiOiWRRnAEJoxDmWkGH3O/o2GPTgZzZVUo7XElCnAL9cA23caqiz/tGU7cEnw65opbe+ZfBMSzwSTo1qBTmRxE8qIVhbEKtE0F2uwEp1os0hpbUEGnVhp04pNQ03GLUoQ1KLB33rvMpiCMjvRjD5cjAdxPvpxEdLY7Suy/GO4H+swy7dc3yysRRHnVnV+V6OjqkhiolzRsPgursCF6Md2TEMCJduzUkXi+eeAm24ytulWbN2G76AdBzDF97WDwEyj/DVci6mWsWaORWtUvWp6b1MJhPWcdyGNofM/iR9tuAaTdgeXk4saGoDp2IoZGMQ6zAIAtKgJ1cSIUb+oEZGuIhKL68AAVSOKKmUIaRpC2vUZ20/KKF/D1VK7WaslKgFxlhhWOYpExZdx7Kczldn5q0Tmm/5jUWUOkykbe4sjLlcF+bSYXqUETSroTvR7e9BaeH6ZtqJpVG7N0NxGw02A52s8EkiYjqYdo1jMOaEd/YFOHF5GS9Ui7ef3WfTYProynaey40jpFTTSfjTYPguUWjSRIGrlj6168F9muRGUgGoWw7DejyGkqAdZascAlcfgfG1dq+rKxzVGDAaOH1eB7m7pyVU04Zag0SuSgQTmMbqmEc1vKUpPItbFkBfYUotFoZYLz3J0KR3XEUYWnzDkzcoAHcYEpfvNoocWp3K0LlskemR0jkR5Y8ca8Z2ATj0Izy+TAIO0wpCJcvbjaI+lWpcyID0niMpsFGg2CrQXU3z91vpRdzcR6TrphSLd2dDFDX6qN3/pWhZzoxFkXH4dS2zv2bZkZkyOc9NVJUxVgYceeogOHDgQ/GIxYjhw6NAhKeKqERHV0sK7f/9+NDU1Yd++fWi0pg3yg4EB4MILfadhZEH2WluQQRdW4v/rGsb/XSGXAm4HWrAY38R9mIcZGMTHcD8uxk9sR15DSKMZu7npDVXhvB/zWNE8Fos6A8UsFNHUCNy/f7b0bxYhh1VYWM30hACi+TdiGa7Gf/k+6tXHT8S4I94ZflSwGymcjD1K96QjiXGW9LnWo1HWMb5fDDekMf5geONPhDJGxuZoCrxHjd1IIW1xe5GZYwjG8fSp2GFLGqBpRsKVN/5pDd66rF2YkSnMeXEswVzQgtz7LBSrR+wzsRZrIT9/1Qt2IYVTsBOkJbF6tZGsJyj27t2LX/ziF/jABz6AqVOnQhPlsY4RQxJEhCNHjmBoaAilUglnnXUWEgn+CjR2iWupBJx6KrBrl/d3Q4CbAFbS+C1bZqRBkoSZycXqN2r60z2AOVWfWT9ERGah2ow2dGEFAODb+HdMRTT9VwawFW14PV7AvfcAH738FEw8LEcez0UBRe1cEAHtGMA9WGgjbTIwFv6TcREG8CucJ/0bIPrFfh+moFEy29POmR1oWbcaAGsDgmqWrwRKuB5fxnW4HU2QTGvlAGkaXsHJOInUSLWvuiz/fawvfaoEkiqlA3lXJiQAOC1TwlOvno6Je6JNuVtvqAURL0PDVmTwerxQ3TAswCqsgr/89KOJG5DFj9puxIoV4ZBWwFi/P/jBD+Kee+4REosYMfxg0qRJmDZtWjX9LQ9jNzhrcLBmpBVwT5jVV/aLXzQSzjtzd3LASl3ZjF3owkr8GjNQRhL3YR7uxxw8s/gbOPObS6XbuB9TmKTFTOW4DFl8BV+oTsgTcRg/wcXS11eBBmAVFqCMJN78zBpMPCwf4LVsGfDs94GtW4FdaFEmrWb9KezD/8PPpL5fK9IKAFNwoEJONCQ4AX0EDX9PtKK87rdM62cChDI0rEAX7scczMH9yOImgHM9GWhESNUoIG6skVU/gXB+x5R5mmOSVjM1qxlkmthawuQQrOsi1KW1trk51DmffRo1kuLWhDVIMaq6w772cEMa5/70C7hpFpB0ZxQPhD/96U9obW3FiSeeGO6FYxzXSCaTGDdunJwVP3qvBTtC83HNqflN1ntx6j4CROuXqN3jqzhR4MNrBIpY0wKGqXHIu59xGKaDKUWZsFyuqk+t2gfOMhb90soV32fZALXZKAROelGL4swQFU5f1eqZqGWBUmpXczP98vOPcNOVOmX29isK5teqROHvXL1eNks0PEyH0uGl9ZbVhA3L376mJZsNY7l2IWo5yxgxZDB2bf3T/O2CqVKsKLO+WGMkAEzHFszAYPWz5w6q3eNEvMbdxZsyMtuQwVysAQAMYga2IBPJ/Zv3sxh3YPIeRevQtGlIJoFZs4Cz24NZOwiGf2iZ0zNluMfDaOPvyQw6sBp/xVlS31+C/0Ib6uvYuAwNm5HBOSjga+iqWJjCH2lBrVayz34ZstiG1oC1cbBrF7BhA/4Tn8MKLMVPcDHWYjZ24BTk0Y5Wh3V1Cg5JXbaW49qsK8jzIAAlxyjW2tqAfB648UZg/HhMunMlNFDge3sFTbgI92AWiliIHGahiNfjBaZrhilFZvx3cDvp3fh44Gt44iy5uSNGjDGJWjPl0HZsuk6HW9StTGWA9jmkYwiGNageLFYL0Vf9ZxI6HUqHG81qRKePpEj8wUcHIo0Mv2vyp+W/r2mu8Fd92IjoDfpsWBazektNWgJo74QRq3iUFvFo78OeTW4sWINF46YM0PprBmh6q06zUaD9aPAcM6pjiqUM4TU2vQXz1dpRL+/Cg+f10qbrc6QXiu5Q+HzeV/9a7/EVNNpOnmRLWElmDuHE6PsxDO0rBmKLa4x6QD0ZadSQTOKEO4xdMEn+REcSt+M6NOCQ6zdaZS/ttAmRwvXDgFVhoIQkLnllJShEW5X5wL+R7MKGa1fj8j8uFUYmB8Ui+on8l4mA9nbDf7lk+LUO/iaJJaWVobRvD1K2fw+hJZR7D2t8JAA0DQ9Vre7rcTZ2ooX77Gs5LlWwJ9FcDRqbgcG6swarwFTfOG3lNbh9ObD08n2YjINSv1OFs4+8xqbZNhbK0JTHtgbgACZjF1KjOrb6Cqfg7V9ZiNMvm4U191scNEsloLMTgL/+NefyT+Iu6BAHfyRQwkysxQKswkysRQIl3Id5OB0v4gZkA60Lk/CaUptF/3Z/X8OhVBvWlmaYU2iMGMceas2Uw96xPXWRnB9gGaD5uMcjbaLbD29niOlOZcoii8XVutPfI6knWS8WFlZecaWSyRDl85TLmTnXg7fnJWRoNgrVlKUfx92h3GuoOqoA7UcD3YruMZtC+KW5V9PFFxv/VNXwrVXx87zOQYF2nljf1uOgpwhmUoko5oWf48NS3+tELy1AjmahSEnolM9XJvtiMVD9L1XSLnt9lWVZNVM2j/YJgshP3HrSAVSn0FARW1xj1ANQ6wpDH/iSQVqmcLvKJLEDzdSOfmpHPx01kh8qF9UFwBSMdpYLkZNakFSJaxSTayiEWNOINI02LesPdaEwF8UeZGkHmkO5XjsGgmUq4vRfPRzb+ilmtiggHHeHsPvBmvFLJeAmi55R79talDI0oga3O1XQssPDCFAGXPPsZmToynTe8BjwEZBbhpH9ajYKtsBXXhlJDMMaMxrdiGWj8Dyc/dhM9+N8GtLs2SJZwWWaFi55jYlrjHrAWD3BG4FkkNYDmIOP4X6lSzdjN/pxEcpIYgHuCcVtgPf7MoD9mIx3438wDkeqnydQwmz8EndgCQDvIzLVo8EoUEYi+LXJ6Kl/+OZiqaNm2eeyAkuxCovwRSxDSwANW3MsnIYXsQvNWICc72s5MdZF+U/BiLtDM4agQ16Px/kcrUkKVPCa4Ch4KzLoQB5X4c5qHWEj6DwxmtBAwEFvdwhVTPVIAqIBSDqk71qxDd/e3YE/f3EA2LFDqT7zGVyF76KIc20SVywkUMJKdAKMhCeJyhu/DDcrtSEMOPusBbvx//BzXH/SN6WCy7q6ELsNxDimMHYTEJgolYDTTwe2bauSHSvK0PB3LYPPtv8VK1a3Kgvum0L61+BrGMB8AGoL6V24FD/CZZiG7TgTz1W0NmHT7yTHNXUksRzX4HG8P7SMSFY464sRHDuRxlSGRu/xikXow2uYqJT5rAwjgtuq2+vMOiWL8/Aw3oancQaex9/wevwRb8dU7MJ2TMNgRS8ZQCXr2NVowzZhu/YghQfxb7gMP5a6DyCcDYg5S4zG+0qaZtijRxllAEgkkSirsa8hpPHvuNNG5pyauNaxMJYyZBE0bHEkShChWDRUWoIi9PU7Rgw/qLWJN5Kjhny+kmPdGTVuHPncgCx9EdcHPOZq8XVcvdDhs8ryn3IeBVmPiqPwpRqrR9D1fM/17Pc4GmUHmitH8eq/Nd05ZqJIs1FQfs46ErQZrbbPN6OVepCtXtd6bJyATj3Ihva+7UALzUWebkV3oHFnzl8HMXFUnuFYnScOYCL1Y57LPUDkuwrUrz+2qPBcy5wllwtnqY1dBWLUA1DrCqMa+PpAnrYl7ZPSTqRpCOm6m1gS0OlcPEw6NO7iEHXg1FEkIxXU3oHm8K6fCF+4Pi7+izF2xN8JQgAXIFf95zgM+9o0sjaD1n/vQHPV790kN3ORD2W+sG5WO3Av7UGTr+vUw/w11otJTL18V+ciPybl56zviqiEpY4VE9cY9YCx7+NawWDj+fhqaSnW4ALcjU8gix40YzczxWotQAA2oxWDmOH6WxlJvB2bkARxjwCjlKhKABiHEjRQZMkXfoVzDF+5MFCuhxQRxy8IwA404+PowywUsQD3VD/nIcjEYqbYnIs1+BvOwFQMKV/P+e44/z0Vu3ANVmAtZuNFnI65WIP7MQeHMTHwqP17JVHBXKzB13AtTsY+X9dhzV/1mDAjKKK8n1Zsw2q0405cBXB9V4EV6MJ6nF1JyDJ2HKm80tFqGtDWBsxwL0MxYoxZjH0fVwD47GdRvv1rRgqBCgij78e5C2lchTtxP+bY/KqasQt34TLpDDhjEVQpfglMPTw/v3C23U+e+3oCAViFi/AzzMF2TMN6nI3teB3SHsE2qijD0NbNYCv+Hx5U8o8NVq9xF8twE76IZYGutQMteB22Yw7uD9x+3jvg9904gAbcceK1WPTa99CKbWN6TKpA9v2bhSJS2FN5bvY4hHoDARUf1xeFPq6aBqxeDcxzx2z5QuzjGqMeMG60GxAYn/0s8NWvelpYgmInWtCMXUqTmTEJtmMP0mi2WE7qcTo02xRWvwW1GI9V0goYqgpJiy17KzI4CXsxBQfH7H0twr1YhHsBADvRbBvPfsAiX2Za4r/hDZiI11AL0mrUSyhDq6b19APz/VmMOwCAG52uAtFpjBfM0bca7fgL3oK1mIV1mIU5r92PT+G144a0AvKbxnPwS2RxEzqw2hUUq1fe6bDe36Abcw3Ad3GlkLS2tADf/nZ4pDVGjHrB2CauR44AX/sagGiJzi6k8GncgX7MV5pwEpWc2ix3Ba9rhE0kvTBWCZUMat2XSZTRhV7sRAumYghDaMEb8DyyuAlUIzIWJpz91qyozOF8Z7w2bqNhDUyA0OxDvcCK29CNPDowE2tDVwJRxVa0oQsrbBH1c7GmagWO4caN+BIuxw/RiZU4HS+6Tsn6Oaoyfkio+f0gBPZkvFL9b6dawh8bZ2Dr1iTGixOExYgxJjG2iesdd9REoO4BzMF9mIs9SCv7zLImJZmJqp6JpFVbc7TbWa8uBdPxEq7FchuB2YU0AAS2Vo42ZEklb8NgplYuI4EEw4o1msR+N1I4GXuU2jCkteA/6JvIV4jN21LbEZAD+8bP8RF8Ff+fTeYJEGuUAvX7HoUJGZcBwye2o5qy2IoLMYDv41Nown7b50Etp35diZZiBX5diaFwWohfKWcw/sGVsbk1xjGJsWb8seP552tSTQHnYgYG0RyyT99YRD2RVoBt/fD6Ti2wFCvQ6rC6pbAbKezGD3AJdo9yPvioYRIhXt8ngFCPXsPCI/gQNMjbJJc19uJU2l4lrek08J5/mxpZ+7zwUfwCKexxHSHPwKAwkUe9PQcVeD2rMjQMVTaNXt+1BmslLHrCc7EGvVhqI61hvb+vIOXrdwQN38FVWI0O11xz0sFtQEcHsGZNGE2MEaOuMLaJ6xln1KSav6MV07C9JnXVOwhapIoHflCGW/TdiTDbawae8dqiI8kMTEtUyidxd1VP4lglr3y9jPoEAXitOYM5qfUAZFx5NGxGG760/zM2kvjPu9fg3B9fGl1DJbACnTbSBeCYnr/KgmXM3Br9B76FPRXy6oUECNOxBZ/BN5BAqepi4SSHYeHr+IyvrIwJEFqwGxrDil5VdInTZsU4BjG2ieunPw0k5dNJ+sEupDGIGZ6yI6OB0RCJStYh1YpyEDvvtmwR+XL2v7lIjkPJs02pMe4uIEIR/1yTiOwwx78G4MTzz8OkPVul/M8JQBdWoIwkEihhJtZiOZYij3a0CrJwRQ0NwHRsxWfwDSzAKszEWiRQqsv5KwyUAWzDNJyDAr6GLuzSWmx/N9L7rsYutCifmK3AUryI07hSWkG3ZmUAm5HBFfh+IFcN7u+IgC1bgMFBn1eOEaM+MbaJ6/jxwDXXRFrFSlyNMpIYxIy60fgzjr5SSkeaxzqiGsjOp30Ak7EHaWiMOrcigxXokrpuAtFq9Y4mJuBI4GvIHP+qXo97zYYG44z/hz+UutZupKs+kHOxBi/idKzFbFyDFcxxMRpYgaVYhUVYi9nYjlMxB/dhJ1rqYv4KEwkA07ENZSRxLQy3jdkoIt+RwywU8Xq8gPswDx/D/b6u34ptaMbu0J+psenS8F1cKXThCAXbj11re4zjE/UwxwbDbbcB3d0uy2sYhG4vGvEVfAGAkTTAlMsZzcnfmPAID+BjxyzxsUL1OUZphS4DaMRBpih8GcB1+Cq2IBNhC+oDomdCAD6A3wauw2tcD6EZy5CVvt4+NGK346h4F1J48f0XAYcOAbvlLeAX4t4qafV7hMzqQ5NcP4DzfR0d8zAVu7AUX8dUDEGrKJ2otk30eT2gHXnMxFoQgHXaLFz1q4VYh1kAgB7cjKVY4eu6US2QW9GGDqzGX3GW72tIP4/nnvNdR4wY9YhjIgFBqQTckj2CQ1+9A6977Xk04AAuwd2+j7XNX83HAO7DXJcsSi+WjrrczfEC1SO0qKWveO0xMhppdelKETbqIQJ9EfpwLxZgJ6YiLRHC/3HcjXuwyPYur8fZONByBk4cknuXy9CwFRm8Hi8AAF7E6WgN0Vo2hDT+vZKwZAdOCT3BQxDUQlIujEQdW5BBJ1ZWNxYrcTXaRtF1g4W7cBmuwPdQRhIzsRZrMdvXdaTfwxCzEMQJCGLUBWqdYzbsXMf5PFHaks7byEkdPAd0DhfRXORpMzK2z3chRTdgGc1GgRbhbtqDJlce9LiMTtmLKfQDXDrq7YhL9GUmigQQ9SAr9f0daKG5yFMCOs1EkRYgR53ola6vBFRz2gMUal77H+ETNBsFSkAP/dphlVrMcbuQCtyuEjQqQaNb0V3579HvO2fZh4bqs05Ap83IUAladHVqGlFbG5GuB15vw16/Y8TwgzGt47pmDdDePvLvEa3C4Jg+rYTV291i3Wnswc3IYj8acAQTfOchDwuE0bd+1QuacACX40ej3YwYEWMHmpFACQuwCuvxQeyq6CuL3oNm7GJmsZOFBuA2XFfV9vTrM8nCizgNp2AnZmAQg5hRlwoAtZhj5qMf78Qf0IvrpH/j1gAmlAFci69Bi1jbwu/c24iDuB5fxpdwY9UFbTU6oksNTTQSpDVrVhQ1xIhRU4xZH9dSCeh0cFQvrUIVfHD7aqbMiIkpDF/H0cBoklby/kqMGKGBKiUJwq9wHlZhEX6F82x/5yFRITF+31kCsBD3IIES5mENunz6TLJwI75kC6R6I54J7dpjBQRgBn6NnTg18LUSMJQ9op4bg1y/E1+3SZbtQSr6xTgO0opxjGDMWlwHB4GtDte014XsyySamI53K2e9k9aoLNFj3cJtBq/dhcvwKfxwNJviG04pMfPfXs+F93eZZ2pEr2/BTKzFClwdyhgw3yHrtaZiF25CFq9iAiZhOIRaag8/74gGIIubsBPNEbRIDqznERWasRszMIgU9jDT8EbSlmnHpiRajOMPY9biyto8TsVQ7RtynKLeFQ2Cts207jk/G+swopnzuArfqxt5NxnsQDN2I81N7BAEKj0wC2vRhm2hjC9R3RPHKGkFgr17zdgFwuhoVNf6TejAvfg2/n8ARyM2rPaUoYEybcCMGSFdMUaM0cWYJa6szeMQWtwfxojhA7XIwFVL3Iwem66lSN7Ni6DXmsD/Fxbje7hSqKdZq+eSCIlSiYiJ+bd63yhF0b4ERiy2zp6uRX94jaMwCfUSfBtTMRTqIuzsI/PdfurKFZEn64kRo1YYs8R1xgyg2XGqtA2to9OYUUS9L25jGeYi9iNcbPv3aOAPeKfv3+7DFKzFLAxihi096X2Yhw6sZr43vHHF+jyqMWimz12Cb+ILuCWiWtSw22deeT+o901SVO0zk3OM9uLkJoH1/0ycMDOHbToruBRWjBj1gtGeG3wjmQQuvtj+mZHdqvW4InNhTqT11m+s4/rRwH6cNGp1E4ASEliOpb6v0YQD+BXOw4s4HXOxppqidAFWYQ9SeAOexywUsRA53IhlKCPhaQ0ME6znbJKEJKLPs64yxnZi6nE3xwBGH21GK5bi9lFrQ61Jo7O+sUBaTUv1LqRwDgrVE5bYvTXGsYQxS1wBYM6ckf9OoIQZGMRqGPpYYS0sx9MCVW/Hk+ZCMdptmoSDo1Kved+341qk8Urg67ViG1ajHTtwCtZidjWS/W84AynswTAm4CZkkVQ8ELUulqsxV7ldB9CA/bCLmZuW4ajJgmog0X/hM7gHC+pmUyWDsNr5W7wf/4Rfh3S1sYVX0KQ2ViZMiLA1YiQANGMPykiijCTS6di9NcaxhTGdOatUAk4/Hfi/W9dgBTpDz2ZVyyhTURvGwk7/WIVJUA6hAZNxkLvTi+I5lZDA7bgWn8NtWIifIIeLvX/kAdaYNmnqHqSRCpiX/QAmYwoO+WrHTjSjDxdjM07DigAW5qhBAH6Kj+FjeFCZ5Mc4zqBpho7qKGAhcrgHC5HNAjfeGM4148xZMeoBY9rimkwCAwvXYMBnvnAZjKZVZbTqLldKP9rHjFUpKpi+dlMqpJXXHwfQEHrdl+IH+G98FAuwKjTFDNZRfwKmlSYYaQUgRVp57WjGLnRhBU6rpFQNG2UAm5HBOSjgG1gS6FoX4IG6IK1Rt4BnWT7e5wUppNNAa7hxFyr9vgNTkU4DX/hCqE2IEWPUMaYtrqbJlbZujcwqaZC4JBIoIzFGpmsjp3orJuFVpLHHV9+UYYq9x7DCaVndhRRWohPr8UGbGH4Y2IkWG2EtIVEXZClqlKAhGcG7RgDakcd9mIfZ+GXoz6vWKFece1Q2GyonA14nTl7XKkPDbqRA0DAVu2SbeGxh5kxg3TqJvpKzIh3GBGmptC3IYFv3Srz/tvACs2KLa4x6wJi2uJpZCFSJmcrSP5KFhcYEZTDaSMhjHk7Ea74JfQIxaWXB7M8jU4zgh1OwE1/CjViHWaEF7ZibhmaHlVUbEyMwOJIg5X70+n4JCczHAO7DPLRjNe7BAr/N84UofGJ3I4UL0V/R45WDynzgFYgn+ptB1Ahfx9Xowgr8AJdHcipR95AirfJPxSStMuMpg214/+0dRm70GDGOISgR11tuuQXvfe97MWXKFEydOhUXXHABnnlmFNMTKqawM1/0hOPfMqgHeRYZEBJIAFiKr6MBr452c45ZjD8wEvwAGFb5O3FVYMu/VXInbKH9sQQ/mZd4MJIWlFFGArfisxjA/NAsgLJziEnawtImLcNIX2vV4x2t8yBRvV/EMuRwMT6Ju9AQUZBjvZ+DeY3lITRjJa4O/bqa2TNdXcbpZIwYxwiU1sJ169Zh8eLF+O1vf4tHH30UR48exb/8y7/g0CE5v7bQEYLGR71PeqoISyC9FjAti35aHOZz89uGabBvnP6Ks3zVbf93ou6zktUzWOPCVMv4Dq7EZ/HVGrfIGFsNOIgDaHBNuAcwEYcxXnk8G9fRsAJduB9z0IE8diPNrV8WYb1XrPEb1Zge6+/KtViOPZxnx4P0PRMBW7YYp5MxYhwjUCKuv/jFL3DZZZfhH/7hH/DOd74TP/zhD7F582b8z//8T1TtE2PGDCCTAUm+xixdvrE+6TlRb/djHmk5F89ypfe/imtRVrQlmkdrYR6/+rFmbsc04b9loAH4Pi7D19BVOVKUoxm1zCpUj5s7U57ICd74TwBoqfh7h/mOyFzLFNSfwrA4NuIwJuKIrzYlQJiOLZiBQdyHeTgFO3ADsq4kCSpjW6UdXj6w9TYX1SvuwGLcjGXRvmeKp5MxYtQzAp0+7tu3DwCQSvGzyQwPD2P//v22EhqSSfx24coKMYqnyXrEiBarfaiZGV0ewvkYp2jv3IoMHu/O46lsHvsxJbQ2ysKITm/DIOziiEYCjIzyWPw7XodLcTcA+RdyV43TG9eLHb8MYAeacTL2jak3PspNsmn5LyOJL+FG/Du+UxOd2Vpo7B4PmIIDACLuz+eei/LqMWLUFL6Ja7lcRldXFz74wQ/ibW97G/d7t9xyC5qamqqlra3Nb5UulErA/FX8tJUqGCuTJC/LUD3DyIBUxjNX3Y5yXw5/6C2OZHSBmiXgOtyKy3AX7vr2MF742R/RUJn0awkNQBdW2NKnArD5G6o8kxvwFWn1B1PofxH6cA4KWIgczkHBJeAfJkZDS5hHvAzLZe2feT3DaulPoIReLD0m9J/HevtlUZOTv+9+N/ZzjXHMYJzfHy5evBh/+tOf8OtfizOpfP7zn8c111xT/ff+/ftDI68VUQFsxTzcjzmYgUGcg1/iRnxJ+VpBJo4ytJpJZe1CGhoIzdhT/WwIzThlDMjNTHh1LxIfvxZvLwGvWw5s2wacSWqWgK/ic8azOgDgiSha6Y3f4r2Yjs1YhJ9gG1oxiBk2ErsHKTRjt+d1VBNcGEFGRlacAj6MLWjFnbgKb8HTaEKIJxkO1DooTES6NEBaDuhYAa8/TNm7BEpYgFXYjmlIoBR6IpaxBGdfHQsEPhRs3WosmLNmjXZLYsQIDF/EdcmSJXjwwQfx2GOPIZPJCL87YcIETIgo/Z3VbaeMJNZhFtbjbFyPW5BEqSYTFln+NwqUAQyhBddiOZqxG6fhRXwcP7F9J4lyXWT58sLBA4YdMpkEFiwAvnZ7CVfhzjG3uHwAv8cH8Pvqv7cgg06sRAJlDGA+8zesewx6zxlswxexLOBVxh7G0lgJC87xY2q4TsJhmx7t/uNRcsoCNsE/NhU5lOfN2M81xjECJeJKRPjMZz6D++67D2vXrsXrX//6qNolBZaowAfxG4yD/yOR7+MyvBl/wQfxW+nfRDUpmhPT3bgEt+B6riUlXbG+1vuC/sZffRu4eQJyvz8Lv39wGmaihDZsU7pGPabfbcVWrEZ7VREginrBqLven7dfjPZ91dtGitWW3RWrfsph2WcFgB2v6EIvWrEN1+F27nfq7VmrwFTLkG5/CCo8MWLUA5SI6+LFi5HL5XD//fdjypQpePnllwEATU1NmDhxYiQNFKEiKoBt20bSQav6TFpBAC7HD3ETluH/YCNOxLBn1pgwJz3nJGT+93VYLvxd2BNvVBaKEw7sAZYtwyIAiwBX9HO9QxSxbmqFev1WdaE8Vq1F9Yx6JTI3owd/xluxA1PxI1xWdR2xol7bzkLUpHFn5aTKy/UkbFj9s6N+d6Xar2nGQjljhvd3Y8QYA1B6r771rW9h3759mDVrFqZNm1Yt9957b1TtEyKZBFautH92JvxHT5pJBm5GFuNxlPu9qBwD/GapCRtRTbbOezjZ4qc71iH7fFSfY0xaY5howxZoKGMe7kMbto75sRH1nHY2NoxKP72KifUjtahVWrFihbFgxohxDEAjougcNBmIItfxZz8LfPWrQDtWV30M62LSiOGJsXxUFxXiPolxLGOsniLIvpclJJCoF4HGtjaDtM6bF8rloli/Y8RQhW9VgXpBqQSsWgW0///be/f4uKpy4f87M6U32qTNpWlJgvUAh3PQIyoesHpy2kpfkaNYmoZeQAVUeFGqCcVwAAshoof+qPQCAl4Q0V/tPSl4vKC0pCUqiiIV8IKKQJPaNknTJr2RNjPr/WNnT2cme2b23rP3nr0nz/fzWZTM7Nlr7bXWXutZz3rW87CZDSz2x2BRAPRRxEQOu+4tQdorGa+E1kIWjhUQJZKTrbuXBK0tFKDCEcIqdspGywJuCq1u1qXZ+0b84qBwzhx48knRtAoFRxAXvkm0t8O/d7ayiQWBmaj8jgJOcBrgfx+xbpIP374itGYnU7vo3z3POwPhm/kI4/kuH893MUyj1+l9sXqUyu0dcaN9jPp1EPqBK1x4oQitQkESeMF1354oa6jPdzEKihBQzgE2siDQAo5VUic4K/5Vc83Xy8nVT21q59nN2IJfyPO+es50TOAYtbTmuxim0W0391DNNRO2cGTiGVl/kymYhBd49R77jpkz810CQXCFwAuu/9LdXhAHFbwWXszwQZ6yXSa/PYsZ7E6kfQ6EnXXiHkGucyc1+745GGOSiRzNdxEscxav8r0jtcw9/L2s1wahPYK+C2GIaFuFAiXo8h7vKPenU2UrQkQ2oTVfAkkpvbY7SMFNAhk4TBH9TLDdTiFgkkNhTINo2lGQQoMFgvjsr3IWABV05bkkzuCXNjAaQ2yP/12F0TaCkErgBdfw1Cn5LkIcXQDdwAIUIdNChO6Gy0tfg5mIEUwBKF9U8g8mDDl+T623kWwGYBZ9EDJbT9I384fen/egmQjsRZzaO0G6hW8uY0d0irSNUJgEXnCNOnQeywnhQt8S+xUzWMPnUDarN5+Co+7UPOgdw057xrCn/Q4P6Qx7KGUPmUMgu0kQtmTTYabsMUJ0UxqPGOQFQTS/cBO9je7jC4SJ0k4NHVT5xflTzqRrb7cXoHrwEqPoeFZrVgHdlNKOBBwQCpOgyye8snOfI/cJobmA6pnxkZwHqdXcxE3cb9stSibtq9vYGSStfO4WMWA/pRygxPbWcyjl30zXpBJGUc4BHuFTdFGedH1hTOn5RX+TvsM1Ga9zst/FUv4VNELAmXTwIz7EJup4iouBwqgno3c1hvvv8QSOOX7/fc5MjYLgOwIvuB7f3W37t6mT3ET6KX32h0D+hY18558JXSO8noVDJhGhYd8n/usFYeB0jlNKr+26c2JyuotmyrDfJ70iaJrEXkpZwEau4bsZ28nJ96aTau6lMa9a9HT44TDnh9jGfB7nk3wXBZwccqFXaNidJPMlyOteYf6luz1PJRAEdwm84DruzPLsF6UhdZLTNZ12J798TyRe0UspdbRwJRuoYwt7qEz6vpty/peP0IP9trHDBI55ml86Av9S+ZD7WUIP5Uyhx/W8FHCIIs7ib9zKvUzndb7M7a7naxU/jTdhVMYw2SMNfTHfM7QDlA/8enBZEHIl8HPsuR+ozH6RR/hZS+okC9jIVrQQglupZTqvM4s2VtJAF+VU0M1H+SFT6KaLUg5zuuODt58mbTilXQlKH/BTOc3YFd/Fl/goT3hSHs3LQz+3shyAuTzB9XzTk7zNoh/o9At+6k9+IEaYFTRyPd/Km3b856/K4SyhMPHT2GeLyKwajpVWFYR9lRfkGulGASUcTPo8RoQSemlgzbBt8jIOMJGjjk9sfpoovTyY4jeB3Ql6KAXSP1to6Nur+L5HJdKo537q2EQL8yl3WNNrJMzYadsDlPjmYJQ/SuGPdyRMlEa+CkAdLRxmgqf5xwixvWknrZslmqRQeARecCUSYfwnF/tm0PQSOwN0LvWk//bbfJI5PMlstrOI9cxmO2v4PCEDPwrB72DZ6aQy66EhpyiUft5LMVexljto5k3GZjXRCQMVdHOE8R6VUFt0bWCx4wdzuijjq9xMZ4rtrJ08nuAyQihZuPsMbdxTrKaef+NFJnLEU4E6jKKZu5i1qILoluBEZhMEM4SUUp4uUPv7+ykuLqavr4+ioqLcb9jaCvPn536fAJB6Wt7o9PxId+buJbqw0EspZRzIa1mCxkoa+Dk1bKEOLDiOO84YxjHgZtFcR+83C9hEFXtYzU15LU8hcZwxPMhnuYk1tr26FBLa5B4i1LIFamtzvp/j87cg2CDYCrFoFK6/Pt+l8AwjH3/ZrhHcQz/MV5oitPphq9Lv/JhLWUM9VoRWIPBCK5zqNw9zgy0PFNK/0jOWAW5mFSu5Ka1taRDrz65vb93MhoYG55yeC0KeCbbgumMHHBBNVyp+cJUzUjDaRpbFQ3aeYC7VdFoagJy25cznO6K5LOplGf9j67eCMXrdLGQjX+ULaa8L2viYS1CYEEBHB7SLeyyhMAi+4CoMQx+8gzY4FyLHGZPX/P26iBnPm6av1Z8h5PCTBEkAlE1v82gBEjq5esi/rNH3QWp7x9gr7rGEwiDYgquQlhE7OPsMr7a2M22JHmesJ2WwgpW+GUL6swzU1plCt9RbItPEPZZQGAT7vZ41K98lEGzgVy1gkDHaRu+hlHtpZCwDUt8eIv3bOaKO69lHKFVVUFOT71IIgiMEXnA9OXaCDGwBIsapSd2LdvN733CifAOMJpJwpz4m0kQTZ/APrmQ9oFzVVoqglsxI1w5D7v1B71MruVn6lhNcdx1EIvkuhSA4QqAF12gU+gfya0MoWCOM5j7qXhrpxP2oZyGgjyLf2gg6IeCM4UTS3xM5QhNf4laWWz4AZQcjQc2KsCGCif/RBclXOMezPFfQyC2s4Ak+Kn0kV87xrt0EwW0CLbjueqCdUnVgxGo3gjKYp5azhAM0soKbWMUH2MYBJrv6LDv5TyDkmwhDbhMeqs161niaryLxIJV5vG4V0RDbIwScy19dz2c/ZfyhaRM9jfeynFu4nB+4nmfBI/atQgERaMH11Z/LKUm30Cd3J4S91Dvofiwf4dNMYx+raXBVmLiIX7OATezxQMPrF8Ioyuj1NM+gLAv8vJXvV4Haan3Zrd+b+CpnsI+ml65g8/dPcDP35XQ/cD8kcxflvt3RAXizqFzsW4WCItCC6+EJI3sV6eZwfGBoO/8fIXeEvRAwiX6+z8e4myZ6KXUtnncF3fRQxj/xKgcZWdFetFj21rG7kPCzUBgEjOpuJGmIl7KauTxBayvM/cdDjMpR7FQQ39FJfQ9yFTZjwG6qWcIDhHC+jZy63wtvu0rsW4WCItCC6/SP19BFWb6L4WveZLSp64Zv5/fSyFfZqK7wZOIs4QATOMJ2ZrqS3zT28n5+yWT6Hb6zv1lNPZqZhBBUeigFRobwWsketlDHPFo5i1dzvp8e6OFOmtlDVdJ3hyi2fV89lGoDqzmXVxxfsOlj4Le5hgdYwve4yva9bnt2Lq2tjhVNEPJOoAXXWRdHuOX0hwpaI5GrwDE25eBOOoZv52u12sAaT7RoemSYi9lpWJ5cqWA/lexx+K7+JQb0UMIveP+QmURV1t8kIlrT/BMD9lPOZ3ko30XxDN0+ezUN/J3pjt33b5zDdF5nFm2spIEuyimhz/b9Oqiiji08wVwaWZFz+VJ3gnoopZdSPsVjfI6v8Qm+b/meuka4nRrq6yXiq1A4jMp3AXIhEoGPfqeWwwsmMJEjtu+jC71+nKxPjJ7I2BOHc7qH3efTVjXe6+rcaIfV3DRitPMKre3K6OVp5tBBFTexkh7KuZinuMNkmFE/vg8jiTCamcsjXAeMnPYIoziTDoros3zQLx17mUaMCCX00sAa7Kg69JGwiWb+hy8SikSYGd1OUQ5zj86NPMQ/qGQaezmbv9LMXcPKaKUudAOLBlYTI0JnpxbxVVyfC4VAoDWuALXl7RRxpGAG9Rhworgcvvc9fn/NKn54Yk7O90ynMXVSSx0EjXcZPZ6VM5+7AKltXUknm1hAGd38kbe7mncQ+kHQKKa/YMY3K3yeB3PuT4laxzBR1lAPKFsTXzflrKaB7nP/k5X3weHDcM//2ZFjCTXO4lV2MotNLOB6vmVYRit9oHNII7yV2vhnEvFVKBQCL7g68Tb6ZVLQVvQh/vb+T6CW3sz5j91EHVtdy8/J5+7OkzbTytENrzu7X/qVboaxgcWcy59dy0cXMsSe1nuMDh8FnVJ6c35nQ5zSOtbQbsuvsR40pYJulrKar78ym9qbp/P56lYmTsyxgEP3vo5vESZqu4yJNLCKt/JaktAK4hFLKByCL7g69DbaFTKc1DDFiPAb3sO//vg+Qj3dDt7ZfRpYzSza+AEf8VTbGEZZyssrYdLpfJyoz1FEaeJLDtzJmA6msYGFHKTEtTwEY+6iiWOMz3cxfMcqGuIC3DTsKTmMJslK9vCNA3VsbM39tH4IOJNOami3XUY4pV1+gM8RI7lcEvFVKCSCL7jW1EBZ/mwXM51MtaoBCRPlQn6TW4HyxD+opJ0a3sUux+zS0jESt6Odqk83666S/SxmI6VD/mMPc7pnWsBC0zZa5dM8wpuMGZHvRiaOMp56VnMl36eC/bbvk+7w6qf4Nr2UOFLv09jLfqbY+q0+5n6LT7OATcxkB2FOncZas0Y8YgmFQ6APZwHa2/jQQ6gFCwDvt2cnJ5xMPchE1tDAK/wrFexnNTdZuldQVxGDRCijJ77N5TapbeyXLfkg4GYfC6eIjxM46mJup4ih+eos4yAwMvtDJXsCO364gS7IpR5EjBIm7FBIgjCapvTbXMOneCzn+1WwPydTnhhh7qYp/ncHVSybsIa5362ltjbDDwUhYBTEWNcauYIVNOa7GEzmME3czTt5gf1U5Ls4cdzeuo8QZRNX8FkedDEXwe8YLSi8ECLDQHkBC61mtMkFMZA7RKaxLnVx5QRT2efIfVZzE3dwt63fhoBIyrNVsYfHjtZRizhxFQqLwI930SjU18N/cy9XsJmDOTiVdopbWME/85d8FyNOP0X0uxgxShdQFrDF9G9G+tau4Dx+F1rtLh5D5Pd9CeK7mq4vZFpM2V3gz+BZG78yJuKgiiGE0p61oUGcuAoFReAF1/Z26BzanW6hjiV51vrpA2M9K22H23SaR7nGdSfmVoWGTqroccg2TBD8TIwQfTmEM97AQl0EyQthghV21m5N6b+z+pwl9HGiuBwV8uHSSSno6NAmSkEoEAIvuO5JCYa0h8r8FCSFEvrj7lxSB8JMk4Abk8OVrGcvU124sz26KGU3ZzKZXt9ryQQhF3Rrymv5LgvYOCxCUqb3PQZ0U8pCNsYjSpnBjTHEK7OPVLwWmO0+5xN9NSjlY+FenLgKBUTgBdd9KeZF7dTQQ6l/BxBOxR5PPSKQqcy5PE8F3YziBF2U+0IDXM4B/oNf4rdDrn7uM0Iw0R3BAzzMZ5hMf9L3mbatE783K0wVYh8OwuL2Clo5zliO+NUlmThxFQqIwAuuvb3Jf/8Pt1HKAccGOycEPd1GrYcSPsA2prKf+bQM0w5nWu3n+jw/4cNModuwwb2e7LI9ixPlsXqPGKEhJ+7+nSatPlMhCjFBIQb0UsztfIW38TJbmB93E2aGEPBdrqaMA5YH6aN+FZ5sYOVtjHHqPc4Hp3OciRxz5d7Znin9Dl4IqqvFiatQUARecA0nPMF8NnMLK5y9v4P3KaOXGBFiRNhKLdN5nSaaPNkOy7TV6DdhLYT3wqumGWthAZsM28MPNn5WW8lfrRp8rLR/GM32cS2f4G6abG1BFyW42jNLCJjAMaKWjAuCj65gyJ/Yml+M+lZ8XF+9Wpy4CgVF4AXXWbO0f8NEeYjP5s0WyyxnsIeZ7GAR6/kid3MHX/akzMYDG+ynnPEcZT0LTa3qja7x41RxJ80cGDLJMCI2lFbSwCza4iESW6hjPi10UpV0vd/7lVfYbWs/9hGr9OXgmcNO3zmdI7bz+wtn2/6tHfJtghQa9t/Cws4z7Y1UEdu8BXHiKhQaIaWUp3NKf38/xcXF9PX1UVSUu4umaBQqKuDtB3awg9kOlNBduihnCv4K5zqLNnYyiztpppm7LP32EEUUD9ntuTFhpNr6mb1+NfX8L5cRJsr1fItL+CnFHI5ft5tqGlg9LJ63jh43fBp7+Vf+yJ182fYzeI1bkct04eQumhhFlBAxDjKJL7KcyQ7ElU/Ho3yCdVzFF1jBh9jmUi7GNLCKLsqpYD9ldLOEBynicCCEI7cj2KXmpQgl6Xm9zD9GAWhhHKCBVeyngr1M4/Obaqi9wllNq9PztyDYQnlMX1+fAlRfX59j92xpUWoR65TSnH/4MkVBxYb+zXdZUtN9NKgwg6qZZaaub2aZWsQ6NZM2FWZQLaPZtbLFcvz9bqrUPFpUmEE1k7akcpu9zUza8t5GXtVZpt+eJKLms1mBUvNoUbupcrSt0qX7aBiWlxdpH+VqPptVN6WO1rHXbermvaOE1BtUq3tZqgYJD7uv1/XkVv3EQD3K1XkvR6bv3qBKhRlUpaXanOgGbszfgmAVvM7QrY7/2Me35X2ASz+4+3sQP0iRJcFgJm1JH/l50aDX+zxaMl46ebJSNTXG34UZVLupUlFCeX+efKeZtKl5tKgoIc8WYfla8D3BR+Lvbr7r3ahO8l0urQ+EVCtzDcvihzI6VdfLaRwaByp9+0yb39GsfvYzpQYHHZ1akxDBVfADBbG70toKrVvNXavcLYqhHWgYf9tITqKfKjqzXqc/WzldSZ/vxb+uVvQ6/wbXEyZ99JiDB9P76I4RoZ41Q//v11b0hjPYwxrqAeX61qziVH3nY6B6L78y9d7mMqbY+a0CvsrNw+ywvaaXyWyijst5wvB7P495ZumliNV8np9yCTPZwQu8y/RvnbD7tdI/Wl48h0WL4CtfkUBZQmETeMG1tRXq6mD8ka7sFwP9THS1PEEdqM2UW5+IvsV1SUJgOzV0UeZW0XImhOY7diY7CBONH47T/zbDVmqpYwtHON3dwuYBK5PjFLqpptOTgSOE5g3D63dKAfspYwo9pt8Lu9j5rQIWsol/4lW+xLIccrfOo1zLAUoAzUvKIjYXhIBqxHHGUEI/N3E/TzOHp5nDR/mh6WcNAUcZm1MZrNTrXqbR2wtNTdq5j9bWnLIWBN8SaME1GoX6em2fxKzWr5jDdFHGJuabut6ORsSLQdxtzXEmJtHPbLbHBcAa2lnC13zhMioTN/B1Xmc6O5jNeq5kB7N5nenMw3iETxVy38cvmJjDSW+/Yqa/xgixm2q6KXe9PPlE77/f52N5LUcmwsCZdFBDO70ZPGc4SYwQ3ZRyDY8x2YI/2iCij2PjGMjpPn/gXxnPm46UKRtRwpTRE//7wAFNoSPCq1CIBNqrwI4dMHvIkUCYKK8znUr2ZPVgGBvyFNpLKSX0Gl4fI8QBSuLBDIKqUYhhPfqOGfqYmHRKv4MqfssFXM4Tvq0rfUJKXK3pW9F1bEnyMDCPVtZQT3WCCYUiuP0gF7QtzxB1bKGXkkB477BLb7iUbVd8k4c2+v85tfHJfSFSHx37KaKI/oJ/B/TnDdJzamNbaNg4Vl0Nr73mnBtX8Sog+IFAa1xTwy9/k+sIobLaFoWHRBB9gEq1W9T/fpRP+nLwsrLScMu+tihBaAWoZA9z+QEbWeBwTs6S2uH1RctqGuJmA/NoZQt1w+x+/dgXvEARZgVfYCu1tFNDB1WW7fe8XB3HEpIVThSV0fnFhzl5WS1/Lquhg0pf7yCUpAitbpVVHz+KbQqtfq5DI4KoqNDLmziOAXR0pLfdF4SgEmjBVQ+/PI9WXmd6PEKNmYcKoyjnAE3cNSz0aidVLGAjV7Lel1o2P5QntQy6APg+fsGbE8t8NVll06CEUfGt1zBRvsn1hPJgW+lXQiga+SrzaCVGhHUsthzdzMu6VIQJY31wG9V/gLffvZCWj7WyvydCPff70vxFL1Nqnfq1v/q1XIWGPo59jgeShNdUBY8gBJ1AC641NfDpUk07VpmiHTOrbfkb5zCd15lFG4tZF4+i1EO5Z4dQ/IidyVobOPcw9rC5Qy1eYOU5PsB2lvFlyobMQwSNRK30KE74bkGnC3KvMn3IFCT722/UL1K171upZQWNThbVEYKoERSMeZPRaccouwum1dyUZLs/zb9OXwTBFqPyXYBciBBN65rHrMB5Nn8lRoSdzEr4bZQPsN2hUgYXp4WTdPdTaDay1/AY0/gHX+czTOSogzmb406+TNQjkSBKiIiJqckvAqKuzVnF0iS7Xz+g189ZvG75N6kkat/bqfGdkF6IuGWH72d0V29HmMBYF+yUK+lkC3VcPX4LNTUS8lUoLIKtUGxvZ/wB+1pRBTRzV9Kpct3swG8hPr22EXRSq5Nua1P/DmAji5hEH8u53VGh1epzmBEmneAEo01d57eJfAkP5rsInjCNvdTQbnrXxW/mBEFipAmtoD1rBOXa4TqtzypWxhqImHT5JwhBIdiCq0njnXSTSqpBey3GZgd2cdo+zqvJ0elOkUl41L+7jm86Wvd+Z0yOrnZScXq70Uv8WMa9TGMuj3uer5d1oYCfM4OVNOTVljfopg9R4Ad8mJu4j+OMdcz2O9c6CQPlb3bwzFfkdJZQWARbcDVpvJNpANC3Bj/PmrhXAqcqxckB2cq9fsr/cShX75hEv6N173e8es4gCAQhND+UTkQayhXdX+2H+RENQ9HSstFLMStp4HCOwSliKf96wTn8jf/m/2M+LfHAAl7SXwABPcLAZfyIVdzMON703Tu3ac1eiaQlFBTBlhNqaqCqCkIZhopM3yWwipsppde1QSeGdxqNS3jKl6ehs+G3Ab9Q8WO/CA05ofNSaEuth9jQp7/n7XyBr5r6vQL+fz7B/zKXMrrpotz2M3RSzb00ctjl6H46IaCCbvZQydt4mc2jP+75uLGZKzzMzR38Pm693DtNXGIJBUWwBddIBNZk0Yp4G18hLWG8nRCCvv0mWMNKW/uxX+jvRwyHPKXbIEZkSHv2E1Pvj35NPQ+wg9n8jX/mF8ywVb8HKeILrOBW7uUBPmex3LlRTg9308RnTqzxdNzoZwI38A1bfoFz5ejQaX6nxmQ/vlOgPV8ZPezZk++SCIJzBFtwBaithY0bbYcG8VKYzLWyYwlBE4JIkMsuuE8YGEWUgxQ72lfS7XYkChsKcj7EUkknl/MDU0JMankm089GFrGcW9jBTNN56sFSeigdFkjFLPkSulbQyGX8kHEcd2wi0ts6k823AgYZOyIW9wpYyVJ69outgFA4BF9wBSgvx64RT5AGrmCLrZkp3CfzL36t88n0Ofpe6tHjsuHEYRizpMvrFlZwPd8yfZ9eJtNEMzfwdcBbUwu7KKCbUv7EeWyhjhIOWPp9JuE0BDzHv2f8/XoWUky/pTyDShg4kw7OOyC2AkLhYFlwfeaZZ7jssss444wzCIVCPP744y4UyyI5ehfwmlzK4WdBO5uWYyUNRNN0OW8jK/nnMFC+8NqW1C/vXjqcPESZy29DQB1bTP+mjF7upomvcwNHON33mghd6PwMD7OKmzDywZ2JRLd66dzrTWMfC1nHQYqTvttPOVewiX0EwyO/k+9MZVjCZwmFg+Vx7ujRo5x//vk8+KCP/DlOmWLqMn2g28pcvsaN7pXHZDm8+p0XqJR/E9HDg17Ld4j4RFxcyVLA/wKVW5jVQjqJmZZ32kSgN0V4CQJ2hM9SDjCRI0mf+bFvd1JNHS22IxPqgmv6ABKahvFr1DOZvvjnXZRxIw+ylVquPW2tvcJ7jJPv57mzgiGsC4IZLEfOuvTSS7n00kvdKItnzOMJAAaJEE6rA/QWfZIxGqxi2JvMMt3TabLlEQZKEiaSfBICfsKHuJ5v+mbLsBCiMx1jrKE7IH3hEiOcVc/tZNCLEP7pc25jVG9+60+9FHMWf2OQ0SxivaXf6u1pdhwspzvp7zJ62MQCXvrn+Uz+S4+lvINMDDhcVE3xrJp8F0UQHMMPMlvudHXZ+lnYswCf2dG1CKnTut0DF1663woiN/Cwb4RWgD4m0kQTX2JZvotim/EZfFiGgFEuaNutmCHk+33wz2iTH0ro4/38EtBCbZvFTrul1nR4KJ3/lxYbd7NOvvuaTgh48ZOrbR9eFgQ/4rrgOjAwQH9/f1JyHJOBCFKx6qLqUa6mz2Ufi6k+HLsppZcSywNhjAgraGQNn3eucA7hh0H98jxERcrEJA7zOb7Gi7wjL+6BciFGiJ48OK/XySQsm7lOx61+GRtKi1lPj4132W1Sy+Nm+aaxl3m00kyTYxGm/Eiu5dUCYVTRQWVO7bGBhUTn1uZYGkHwF64Lrvfccw/FxcXxVF1d7XwmNTUcK7U32ZutgC7KWMvH+QwPcxP3sZW5NnLLTjGHk/6uoIcyG4ERwkRpZAWf4wHnCpcBKxpeP0xCo30Yv7uUA2xiAcUcCsxWiL4jsIb6vORvVmjN5V650k05dbTQQl28nvwmvCZid5fHDPuZEq+DdLloZiWwifkZrytU9EAYDayhnvttH25UwEI2UdPT6mTxBCHvuD4/3nbbbfT19cVTR0eH43lEiVDPGrT4O+4wjjd5mjms42Os4mbezy9s3cerCUs/fBP2IMcYI29ycQPdXCT1kI2fGGB00t/dlLGAjfwPXxzSFBv3hBjQTcmI2i6PoZ1kr6ITgNeZzt005ew/1Ok3uokmZtHGYtYxizY+yE9tlSlTufRQukDWQ1khoIlmtg4JriONMHCE03kbLzOGAZpo5gClw647xtiM9a73s8jSBtvuIgXBj1g+nGWVMWPGMGbMGFfzaG+HRw7UcpBNfJtPuWK7OCFFmCijJz5oZJP++5jAaUQZx3FPp22v8ooRITKiRBL38H8dKg5RzKShQ08VdLOKpcSGFo9bqDM8TBhCc93kZ02jk+gC/Gf4OpfxQ7ZQR6qIoS+yW5jPHJ5isolx6zDjmcAxRw7zKaCHEr7CHUkRy8JE6aCKSgun/vWyGI2Jel00sJoKzJ1H+BvncNnQIdqRyESOcjdN8b87qOROmhg1tFO0g1nsZBa385Wk61IJoaCjQ5skZ81yu9iC4AmWNa5Hjhxh165d7Nq1C4DXXnuNXbt2sXv3bqfLZpq9e2EerayiwbLQqm1LZdfUGhn765NHqpZJt2lbSQOzaGMejzPeY6HVbfZTzkoaaGAVo3wotObDRnQkCGVjOBkXWnUq2TMkmMEKvpCxLwTFBCJXOqmiji08wdyhrfHh/kq1v0P8J8+YEloV8HU+62g/K6OXuSkCYiyHHawQ0ENZ0md6XWyllr0mfaiewyssZuOIeKeMSH2HKvkHd/EldvEumribNi4mRoS/cY65G5r0dS4IgUBZpK2tTV9UJ6Wrr77a1O/7+voUoPr6+qxmnZaXmltUlJCKglI5pJjN3+2nPOnvN6hW82hRoFSYQbWRupzK5afUzDI1kzYVZlCBUotYl/cyOZm2cLkjdVTPqrw/i9Vkt/8rUFFC6g2q1G6qcn4PE8uTrkxO5eFGamZZ/P2YSZtj932CD5uuX7NtGQP1BpVqNtvUItYlvdvzaFG7qbRczh4mq2U0D7sfaOOh9gyhjP1oP2V5b0e/pSja3BJmUIUZVDNpU80sM/f7tjZH5lo35m9BsApeZ+h4xx8cVLEq5yZLO2kxa9VM2gwH/m5K8z7gpUtWBZVBwmo+m5M+dnJitlsup36rwBGB80rWqjCDqpvSnMvjVrvHQB2iKOnzgyl/5zsdYoJh/UXxTnCNklmANkozaYv/6eTCzmzftFre1LSbqqSF93qusHQ/vX30e6SmeeiKhlDK77TPltGcl/4WlLSRumHKkrTtEwopVV2t1OCgI9OtCK6CHwj+zl17O6FO6xFYnGQK3bRTwwYWs5NZxIgwj1a2MJ9Si3G4zXKU3O2GrXsqiLGZK5jHqVOq7dRkPJTjNXZLodACUnyDTzNIJKctynK6mcsTlmOwZyKX8qRygFLm00IJvUmHcirYR9ST43zZ6aKUldzMauqHhe7spJoFbKKHUlfLqtD60xN81NT1+gGkdmoIE2UmO3gbL5n6bRdlGQ+27aaabspN3Us/lHOMcaauT6WSTrYwn2V8iXlsZQGbLb1X+sHQb3A9YQPvHVuppY4t7KEy6fNOKqlji/nt7xHKArYwJSXAAhiMEaGhVlu9Wvy4CoWF15Ky4yu2df7Yqk7VUji5ZWqU8qXJ07YWq5K2/zQNSu5lioHqolRdwQZ1kkhenq+Br+Z8jyv5nuPt71R7b2N2UtslJje05049737K1H00qJm0qVEMqNlsU/1pNLJOp5m0qflsHqblSkyaljGk5tEytMVeZSmPb3N1Ri3kPFry0j4nCedUx8tojm9rJ+5IGZkh7KYyb8/pRcpVE245VVcr1dLizDw7hGhcBT+A1xk63vHb2vI+ICm8m2A8H/zSpMTt0Hm0OFIuXXAdxYDaQF1enrOf03O+x2N8LO/tY1y3k9MKreDMtrZb/VN/v5bTaFkozDUtYp0CpeazOe3zxUAtpzFhG9xaHvezxFDgTbWXt2x+cvrpeR0vjjBO9VCS9Jn+DKl1pLfxfDa5vvD3OunmE0bP7XhatkybFx0yD0hEBFfBDwTfVKCmhjfL879VrW+wrqaBSvbYvs8hJmY8yWvFB6SyXYrsfIDtLGI9s9nOGuodcc8TAso5QA/lLGRLXlr0dI7a/q1e31ez1pnCZMgj3XcqzTVavwkPO0GeiNkT35lIdIuUSK5eHsJDT3YLK6gc8ovqFXuZRpgoq7gpbT9XaJGx0nkQyMarnMVWapnO63z3mjaea9DMN97Ka2xFi3xky/zk6NG8joync5xSepM+082nhntZ0Nr469zAI3wKcHcM85JOqqmjhfm0DPO64Djnnae5vhLzAKFQ8VpSdmPF9kyDsbF/vpKdAz73s0Qto7mgtAx2U741yna1hl6VO1M+MbRDdEbfJe4KGP18FANqP+Wu9MF0ZfJz0k64V8e3ut1qy5NE1CgG4h9XVWnKssZGpUaFtLwXs9Zy28RC/hgPc+kz+R4Lck2HmBg3l9A/vpK17ubb3OzY3JqKaFwFPxB8jSsQ/chcmriLg0zOd1EA6KLccrz5rVzO9XzLtTI5iXL5/vk+5qVrta0+p1ea8Ez5hIAIMcM8EncFUg/NzKOVv3MWU+h27qDjsmX8+dKGIS1lPjzr2kevv6XcRw3tzKfFtbz+l48wmBCRrLMTFi2Cv61o5e9qOjuYzTo+Zrpt1FAKqew9ze13ORciLuyjaQfoqvgA21hJg+1wqmaZyGGaaeJ2vsIi1jOTHexlqmv5KUB985sSKUsobLyWlB1fsbW0qFhVsl3YIKG8rtT1wxxmy9BNqZrNNlfLFHTNRT6SO7aa/miLVBtlJ/wgD0vbtqnjZVVZn9cP9WFUplbmum5Pq9k8DteC59ImVncMRsouj25nqmtAnTpUaqY9Ev/eTaXqptTVHcKXF7qjdRWNq+AHgq1xbW2FujpCnck2bxFUXrR2mtuaKmayg+9wrekyrOHzpkMh2iU/NWKeYOnj7NNJFXdxB31MzKu2axpaJJ0wUdt2menKHwNOFJfDM88wtqcza8/zY89UwOU8McyeNtMz76eMDiot9WX92RO14E60iZU6DU32x06VHfqYQB9Fpt6l8FC6myZe5y18k+sB9/ufURSsEg4QQrk27p23sYlffqE1+4WCEECCK7hGo1Bfr60vXUaRfsJKJARMjfTQTDMTOWLqvt2U8j980ZGDMek4zmifeOdMj1kflWCuLZzAzIRmtiwKWM9CQkAzd1PM4Zx8zuZaHr2/XVbcTrWFmPSJpDOnCAOj+7rhS1+ycdf0ZHoPY0A/E+IhnFN/ZxXdF2lqvWR65gp6GMebls1MwijOpIMa2gGoIbc2sdqvQps3Q3Mz4N27ZXZMzUYxRygyGS43kUr2UMaBvEyA+lgcw70JWAFn3fd/2fz9Ey7lIAj5I7iCa3u7ZgzmAWYH2BBwWvRNS/f+We03iRGJO/I3k5dupzWHn3KAkqyr9nH4c/DqpZirWMss2qiiM2sgA6cmOyexIiQsYmPOJ+JjWfIMJVxn/HvNSX7p3Bra2qD1wdximHupLU1ne6w/6yd5lPm0DHNs70Y50lFCLwp400aAEF0Lrv/rCdXV2gn0O++ElhaorHI9S/09drLvWNXqOznxHWG85d+EATfP/OsLqQ9+rJzvXS6aV6GwCK7gute7wd1KJZk+oBMKE/tCI901p1zdFHMo6+91wa6BNWzng1zHtwje0ReNEvrYQyU7mcUgo6lnDTBcY6ZzlPG2NEp+QC93ri9cLyWm80utx1N9ZzXn/VuEWbMg/OpfcyyRRj4XFPoW8CqWAjCd1+MRwe6gmcNMcCVfY82r9uloTlq+n64Fd3P3JYlQKCmqUiu13HD0q0Sz9NJc27qPIlbTkONdTpHv8eB0jjnW/9PvKAw/VGaGIvr5+BPz+WWjCK9CAeG1Ua1jxt0+CTxgN+mRdr51aYulQwJvJEToAqVKS5X6xIQW1U1p3p/JTtKdu+tpOY3DomYNojkyd7LuuyhRJwPmomn1xGWmD/F9m2uHOX5PdGZ/661K/eDa7IEj/Hpwx6jcqe6+XDt05sKz9DFBzWabCjPoSeS9o6XV6qXmFrVunTaUbt6sVO3QOOR2m13BhoKKjuX24S7jfm2tfJ3hKjU4kHtAAjmcJfgBvM7QsY4/OKg5PHTIV2ELl5scRJwdkN6gytIkNZttSR9t2qTU4MCgqdPb+Rh0s6W6srZ4E6YTNJyOxqTfz+8CTWrSw52eJJK2PozqqoeSYb4kzQhH+YrSlkueuu/VUQyYfq+GC8CZ+4ebdaKHjrYqnFip2++zcFgEtdPC7gvLet0mt0+wfc16kVKjp9ltpxdWtfln/haEHAiuqUAkAmu0rWVC9jeLYoQ4OKGar7HE1PVObktphzI6LR3EqKWVelZzJd9nFjtoXBqF9nZTp7fTkWpmoGzexwqKEFRXs/ihGpTKfJLaafOA/qFTyEHp/Lpdajs1vJ9fMopo2vowqqvJHKSZu5KiZpk5AJTPLVi7fVA/6PQInzL1Xhnl00k199IIhIZ541A5jDVmqGQPW6hjFCdd8wSymI3MY2vSZ++L2T8QZoUwcCYd3MHdfJPrgML3KGKnLzewisUMj56Wy8G9Y696aDstCG7itaTshh9XleLH9SQRUytSfQvm3hkt6muzNmXUMOmagivYMGwrO59pN1XqyfMabP1W2zIvNYiPXqUe4RrXyqzX++DmFjU4qJk7eLF1GAW1j7K8t5mVpGv+dI3Lfdht61NRoECpRaxzvJx+vl+2VM8qtYh1aiZt8TqaR4vqDCe/G52RarWxblNGbWEUMmrFs7cV6hijXX3efZSp2WyLP7PZaE5Ot0s3pao7xaSl0JKVqHH6PJOqEddTLu+taFyFQgGvM3Sl4w8OKtXWpqJr16m6Ms35v1EI2NRBV9+Cybb9ogu089msQMWDC/hhq1l3qp3LPWazTc2kTV03YZ2aNTRxn6oT57fy9Hpvbj5lqux0GMR09o92Bb98pXjfC7Wo6jMG1f4cBW898IDbAS/cake3km5rrYd31QW6UQwk/Z0o1BqNMfpna8Y2qpgH9rVGZg527nOIiXlpX900aB0L82aakmv9Z1J27KNcjeaYms021UNJVtOcTCGZwd4CPwZqT0RsXIXCAa8zdLvjt7RoZq+1tBhqEpfRPGwSMjsYpEYbylWISLZxtS8g6jZ5dgf9xaxVoZCm+Uz8yo3DLc0si9d7aalSa9dq+RyiyPEJJbntNWE5iIdCdG3pd6/OXdj8ErerRaxTDXw1789lth2NFqBOCzgzaVPzDMaM3SmHIROT0fWJ9ohG3zudjjEm6e99lOdc3163cRRUNyXqTppUHxM8z99+uUMJ9tDGC5jEvpNusaOnLkozCq2gLCsU9Hfl2cYWR+ZXEVwFP4DXGXrR8XXrgVTtSa7bL6kn4EcxoPZTbkuwSxzYsg1obqf9lMcHzKIU+XEeLcNOp+eSEoV/0E62Ox128SSR+Mnl1LZ3U5NsNtkVvF66fFneypyvtIzmYcJfF6WGwoKdpG/NntqlSf1+uACSmDKNMWEG1Wy2qWaWqY3UOVbm1LSfMrWSz6t6Vqmr+J7qpdj2vYy9NXjX3rupVL/gvaavP0RR3rS0+4bGzWwLmMRkdG23weHJTMnKfNFFqfrEBM0kywlEcBX8AF5n6FXHH7IeUMtMzPVmtXDNLBs2OdnVSqYObF5oZ9KlbJOzE9vKqTaWoE3sRyZXOjbx6GYT89mUdeDPJjw6pdVzclJ97WMjS3A9SUTNZ3OScDibbWo229R9NKj9OWoX9faZz6aMpkJGfTdbMhZQSh1xW5fOC0LqNXb7ntG9vGpzK2PpPsrTLji8SItZG/9zVCjFxCQ0qBYuVGrduuFzkFmFSqb07Y+0qD2R5P7Vx3j1Qz6ovsxtqpllcfdqoM2FTiCCq+AH8DpDrzu+Ga9ZZlwMJf6dun24nEZTg7u+JZY4oKQOaPWs8nwA1ieMdJNzrlrKdIJxLtv2B5mo+lO2FdNpOVKTWb+5frO3G/zpNhWrynwoyKrttd+eMfV5Uv1XpgqDvTmYmOiCsR1ToUwp3WJW/2wZzeo+Ghw1eTAjzFpNX+J2UztKTvch/XCb2TMH6cZfK/VrR/DN1h9CIW33z8jleK7Ca1ubUps3mL/HunXOzKciuAp+AK8zzEfHb2nJPAhYFaASBTGzfvWyaTX1dGbloOoMWxcSBwlnFGjMnmxNNxhb2Z5KZ1+aOqjfWGLthKyuUdW31ewM/nb8IGY6fOHVIb19lKvNGwaVamkZOvRjbFO3nEZ18DT/eU5IZwuYTbDQF1Tpt/LtlSUK6k6aVJhB26ZCRilb/0r0M9s9ZPLgZp1bOdGempoxp+F361BVuncrBmo5jfH6zlSPZvqXkWeVbGmQUFxwTpdCIaWqq5UaGEhWnli1pU5N5eVKHTs2zJlOxiQaV6GQwOsMXe/4uo2AHhJmyLinuTn9S23HxYg+AZndRt+XYEeamm6/Pbm4zzZm38pOTetYmPGU80o+b+o+100cPjnrAqLR9qzx6X0MD8ElDuihkFI7m9ss1Xn/JHMa1fKUHeTqai1QQ1ubUtuWWctTf8bh9aql57hADXpgL3sfDaq8XJsEP12a2abu/05w1kOD/rwHbNpOJgrVqeU2657Mri15trR76MCmmWvTLeoSF1Bmd0y83FkxiqJmJm2kzrMyGqX7aDDoL+VJpkBm284oJZoW6W1oRVg3o4gApVat0iKTQTZtvLn7gVJlFtam1dVKbFyFggKvM3S147e0qFjKMjRWVaVUixbaMN2LncuWtdnBPdEeKjUNWw23mBdcY2gG+KMYUMtoThvm0+wztjW1JX1kpB3QD4NkCyuaLg01iTaaVlZmL1dJiVLbtqn1a401qqma1+9/b9Bo7aKRqSNkSKkCu9e+fHWhadUq42dOXBzY6c/1rIpvX6db/JyyBU2v2e9ngupKEUY7I9Xq2cYW1dKi7SgklttpN2ipSXs/tBPr6cPEavanmZ5rH+VqFAPDvrZrm34/SzzrO0aaSzNmMvl296e7I6tnlbqfJaqeVUltEGbQtEC+kbq01yZqO63syFixfa6qUmrRFea08RVlg+qWmwdVXVmyWzb9oF+q/Wq21OKMQwGllAiugj/A6wxd6/jxLdThg0GMkHqpOb0wZWUATE1m/R+m09YMWw0PGeWatZnVfSCmlj81zGf2LUzU8fJqNTgwqBobtY/TawfMhRU1queZtKmXbk+QKLPZcSSMvNsMlNtGgsPxsqr0o7WRwZmJtJi1ca1zrpO6kQCVLfCFXq9LTMg7VrxdpE6+2U49ZzMZ6aZUzWeTeuyaNvWLJevUC6vakvxH6hsiDQ3aT2a56J4sUaOWTWDoGhJcM5nC7KZKzWdTXPBeRrPtg0F2A3zYySuXQAiaCZI77ZO+z58yD8m0pW5lgTaTtrj/7XQ+nhNtqa20qxnb51DIfHl31TYPswMwMvnoNuE+q7nZ2WlWBFfBD+B1hq50/IEB9WZRecaBsHditRoTMnYmDrltOWVL+yg3FOh04/0kLAhWXZSm9XtotPWkH2JIdwJ5OY2qrEzbVm++M3tQhmz5JSZDzZSuem1pGe5EFrTPEiooVXBNN8HECKWpXGXutJ5B0vuLE/HcuwxOlxsJw0Z1es01w2+ZqH01ch+VLum2srUpbRZm0FB7rwsN82hJezpeL/N1pdld8LS0KFV9RuaDf1FC6qDNA1h6ec0KDNnqLt3pfTPvR+p3b1CpdmPdo4YVO2GnBE637HAzafeX05h1S92siVf30I6UGW1n0emnFnBmlRn3s8SUjb3Z8qY7aGb0WQzSjrlVVc6ZCOiI4Cr4AbzO0I2QrzGTBj+p272pW0RWDktECZkOVXgfDcM+rq42lquia80NbhupM5xIjQbj5EhY6a5N1uzd+t42yxNRuq2ztBqMUIKAOTioSabLlmlp27Zho27iLn/WaGf6yQijkXsoSkXMhPCa+Ex2TUr0reYrh7S2enu8sKpN/WHZurg2KJsvyKqq4ZYVRgsC04JGtbaFn3rII5sdnq7BzKYxa9uWfdbcti1bFCr7h39OElbLabR0+ErXVts97GMl2dG6GgnX6Rc9zpTT7H3MmiNk0qhqn5tzT2b2fMEymk2/u79Z0Rb/06obwGwHrMyWwUof0xZBVUljbijDuj1XRHAV/ABeZ+hox09jHmB2QDC7RZTud1YPdTQ3p7G7TOCFVW2m7tk3xvxhlZm0WXb5k0tM7MSts6xaykwCZgqJymjTQmS647QtLZpJgYX+Ye8Qn/EhjtJS7ZFThfFMXhJSDxha2dLUhOcytf22nw3rhIODybaz2YSG/rHm/KduW5am7hPQn9/YiXvVkO2pvX6oC7zfZ6Hj74kT6dHTzdu5pi5CE/tJOgHQzV0ko9RlcHAz2y6CUZ832waz2ZZRW6/b/4cZVItNvrsv3X7qcGo2N4BWd52y3S+XhUbimJtOKeIEIrgKfgCvM3Ss4w8OqqOluW/bGm0RpU4CqQdxdE2YmYHoDarVW6oGTQ8k69dm3zq1GtZxEessu/zJZQJPdBuUs4CZ3OSqZEjJbVqITOPAsKVFqbLJmbfYUzWedurkjQxamHQ+Ho2SvujR/7ZrtvDCKuN61u/tpOD2h2XZnUcmCuOpQowTgS9iaJrX3VRmfKd0oTCXBZvVtOuaVaauS7f4SUx63V0ZWpek1c9mjuCEKyu9/q66YsCUQJ3NLtNsG1w/cZ36zmXptPVamkeLKi0178Fk27K2pI/S7QZk0yanMxvQgp8Y7y7k0g5/WLYuq1LECURwFfwAXmfoVMcf3Nbm6CSSqiVMioKCNiBvrl2n6sraVMQwcpbRQKQdCrMykLS1Zds6Dan7aLD8bFY1rrkEHUisy1wFzFR0Qcfs8wxuaxt2jyFLgWGXZ9N4mq2TVH+zRpel8/FolHRbNVsa55R08nvG9azf20nBzajuU9sh0y2cLMu3uTbjO2XnwE+mlEkg1G3uTTU+mRc/qam+Pvlvs1pXu+YAZrSMVn0tm22Dnc1a//rFF1pUh4GAXEuLWrhwSJDLZts+9EK2bRtePjueI4wObMU3lzYbeb+pzuyzMUvK9q45hQiugh/A6wyd6vgv3W5uUjM7IJtxLr5u3SmhJ3H8M7QzrLK3X6OPr7Vp4l/X0qKuKG8z/ez66v/aT2TX5KZqCtJpB6xoG5zUuOr1U1pqZtvN2M5Sr1+zc0I45TCvmUAMZiN46Y9t1Kf0iS7RVs2WxjklpdO46vVitr2OjC/PWPdHSzObf5hpBye1v/ezxFRM+VyjxGnvxynb3HSCcvzwWprGN7P4SdefEuUfs/1kYKw57yip5wSs9HWzKcyg6gxlfrc7wtVxbxUtLUpFDATkYbaeJl60dPKtLoCbdWOWOp8MK4uRv3EbB0d1G1cz9uROIIKr4AfwOkOnOv4jH2sz9WKbdZpuxqWJLlu1tAyfdN9SNahpABzYr9HH19TBODI0GLdsyj7A6ZPmJya0xM8+fbo0sybXaAJ6qdlAO4A5zUskooUl1Ew67As5RvUD2TXT82hRa9cm/9aqNyw9cMG6ddpBonSLin2Uq/tosBy+UVc0G/UpI1s1qxrnxDp5g2q1fu3wek50UWV2QdBevznDlmco66LNTDuYsTE0u71az6r4Pa8ob1PPfl5z1bV+7WDcvldPVreHU9Px8uq494VMgnJ8rWbQ+FYFwkRT8cRFgel+8tOfDo/ckZLB0dJq9ZZp6T2zOJnMvNv6MJtpATTMhN7Ei5ZJvjVbn3OL2zJlkZ40mWfzKuBUSNdsiOAq+AG8ztCpjn/H7dntS/dRrspOP6YOTsguOJ1ZOZhtFylJtkoToMsxso6v6UbXodQTKlUbFyabKbS0pNfkGk2S8WceetjoWs1Uwort2rZt2QVmM66TUtHdQmUTDlatSv6d2fgDKZ64kurQaFGROIFXW9j1S1Q0m+lTZjXO6Sb7bduy9zOzQoNxwA9zM7TZdsjmcaCbyRlvoNm4RtTPfjiQtl6NFF3pvDUYesaApJOXiYEyMm2VJwkbCY0/uK1NtW0bjPu6zaaAMzpFrvdTMwuRzsiQ9tKkRrKtTXP8Yab9cknZ3m29Pa2+Z2ZetHTj76YNg2pPJHt9DhzLEAQlGwaZG/lx7Urw4+pUSNdsiOAq+AG8ztCpjp/djY42wf70pypjfHddO2R2u9ZLso6vBgPc8Qkl6u/XNCc5fU/9SWr0ItM+ZlXyhGhG86JPcKa1TyZZmxBwKVNZ7GpcUwW8LNWuyss1+pKlAwAAEXRJREFUjaXZXT8LzhQM8w+FzGsGE+s58bnS2fpmaq9aWpLLbXMFZ0XznanvhBlU61iY1T9x6gImXZ0m3sLM6X0jVZotYSpDuVL7WiSStQjx35aVWViIpMvQIAObAegsp0zvtt7tzNzHjkYyXdd+tjFzfT7b6MBkkaIsOC1N5KxcxhE7iOAq+AG8ztCxw1lDmqdMk5ruckgpZUo7ZHa71lfYEBwSf9LUNDzudbZn1idEMxNGombGtPbJBHaFA6cESjPV7uZiSO+r6dxILaM5Yz2bsTFNba+4qYpD83K2dqiqOnWNUd8JhZQ6/XTt+uU0DvP+cZKIWk6jAi3imJk6Lcnimlkvxy+WpG94pxctqX1tYMD8K68v8MxoL9NmaJCBnQB01dWa6U2qaYbVlFh/Ti4SrPBsY4vaE0muTz20sdP4SakigqvgBwIruCp1ytYxnUA07IU2MSC7bQLgR+w888BAVpM4VV1tHKbViYnFjOCVJf6AJxOBm4shvd3Wrx1Um280Z3eo17NdwcPpusnWDtmuSYwklimufTaNq45T/dUvwkZiO2fTXlrB7MJj27bh44qNM0jDkl5/AwPDNdCpKRLRrnOawQEtiIhRaGOn8YtSRQRXwQ8EWnBVyviF1iOJCu5iZnLOVfuUSajORTjwciLwYjFktZ7NbrEuW+Zuuc20Q6ZrnBZcnNSW+kHY8MJkxe77Z1doTdxJy5fGNR/4QakigqvgBwIvuCrljxd6pGJW8LAzwZlZlOQiHBRav7FSz36a8M0eSkt3TWNj5mdobLRWHie1pX7oY16YrFh9/3RTL7vCq94v3bRxFYYjgqvgBwpCcBXcJdvka9be08oEl+7gkNFk6wfhwC+YredNm7JP9l4e+siVxsbhmtdIxLrQquMHbamTeGGyYuX9s2OqYiSI+mkBNhKQ+VvwAyGllMJD+vv7KS4upq+vj6KiIi+zFmzQ2gr19dDZeeqzqipYswZqa63dKxqF9nbYuxemTYOaGohEjK+bPj05z0RCIa0Mr71m/PuRTrZ6zla/Ops3Q12dq0V1lBMn4KGH4NVX4ayz4LOfhdGj7d/PbH8NCk7XTy6sXw9XXmn/921tMGvWqb68Z48moqYiY4WzyPwt+AERXIW0tLZqgktqDwmFtH+3bLEuvJphxw6YPTv7dfrkJVhD6nfk4eQC1AnM9sFUjARRfZyC5LHK7XFqJCLzt+AHwvkugOBPolFtojNa1uifNTRo1znN3r3OXickI/U7stAFu1QN+5492uetrd6XqaZGE0B14dIM+rWrVydrT2trNeG0sjL5+qoqEVoFoRARwVUwpL0981ayUtDRoV3nNNOmOXudkIzU78ghnwvQTEQimrYXhguv+t+lpcmfZxJEa2vh9de1XYJ167R/X3tNhFZBKERG5bsAgj/Jp1ZO18Zks1urqXE+75GA1K97+M0u1soC1GuzEF1TamTCsHo1zJ1rrS4jETFtEYSRgAiugiH51Mrp2pi6Ok2IMrJbS90uFMwj9esOfrMjBf+bhdTWZhZQRRAVBCEVMRUQDMlmgxYKQXW1e1o5sVtzF6lfZ/GjHSkEwyxE15QuXqz9KwsmQRAyIV4FhLT44bSu37ZeCw2p39zxs/s2cRclOInM34IfEMFVyIjR9md1tbaVLFo5Qci/e7Fsiw8/LECFwkDmb8EPiKmAkBE5rSsImcmnHWlrq6ZRnT1bc+g/e7b2d6JpQjqzkLIy2LhR3mVBEIKFHM4SsiKndQUhPfmyI00XIES3q03UpNbWQiymRcvq7tY+6+6GpUu191uEV0EQgoKYCgiCIORAPuxIrdrV5isKnlBYyPwt+AExFRAEQcgBM870nXYvZsU/q1+DEAiCINhBBFdBEIQc8dq9mBW72nxGwRMEQXAasXEVBEFwgGzO9J3Eil2t34MQCIIgWEEEV0EQBIfw6iCjlbC9ZjWp+QxCIAiCYBYxFRAEQQgYVuxq8x0FTxAEwUlEcBUEQQggZu1q83F4TBAEwS3EHZYgCEKAMRu2V6LgCbki87fgB0RwFQRBGCGYFXIFwQiZvwU/IIezBEEQRggSBU8QhKBjy8b1wQcfZPr06YwdO5aLLrqI5557zulyCYIgCIIgCEISlgXXjRs3snTpUpqamvjd737H+eefzyWXXEJXV5cb5RMEQRAEQRAEwIbgunLlSq677jquvfZazjvvPL7+9a8zfvx4Hn30UTfKJwiCIAiCIAiARcH1xIkTPP/888yZM+fUDcJh5syZw7PPPmv4m4GBAfr7+5OSIAiCIAiCIFjFkuDa09NDNBqloqIi6fOKigr27dtn+Jt77rmH4uLieKqurrZfWkEQBEEQBGHE4noAgttuu42+vr546ujocDtLQRAEQRAEoQCx5A6rrKyMSCTC/v37kz7fv38/U6dONfzNmDFjGDNmjP0SCoIgCIIgCAIWNa6jR4/mggsuYPv27fHPYrEY27dvZ8aMGY4XThAEQRAEQRB0LAcgWLp0KVdffTXvec97uPDCC1m9ejVHjx7l2muvdaN8giAIgiAIggDYEFwXLlxId3c3d955J/v27eOd73wnTz755LADW+nQI8yKdwFBEARBCA76vO1xpHhBSCKkPO6BnZ2d4llAEARBEAJKR0cHVVVV+S6GMELxXHCNxWL84x//YOLEiYRCIcfu29/fT3V1NR0dHRQVFTl2X8E9pM2CibRb8JA2Cx5+bDOlFIcPH+aMM84gHHbdKZEgGGLZVCBXwuGwqyu1oqIi37zkgjmkzYKJtFvwkDYLHn5rs+Li4nwXQRjhyJJJEARBEARBCAQiuAqCIAiCIAiBoGAE1zFjxtDU1CTBDgKEtFkwkXYLHtJmwUPaTBCM8fxwliAIgiAIgiDYoWA0roIgCIIgCEJhI4KrIAiCIAiCEAhEcBUEQRAEQRACgQiugiAIgiAIQiAoGMH1wQcfZPr06YwdO5aLLrqI5557Lt9FGhE888wzXHbZZZxxxhmEQiEef/zxpO+VUtx5551MmzaNcePGMWfOHP76178mXdPb28tVV11FUVERkyZN4lOf+hRHjhxJuubFF1+kpqaGsWPHUl1dzb333uv2oxUs99xzD//+7//OxIkTmTJlCpdffjmvvPJK0jVvvvkmN954I6WlpUyYMIH58+ezf//+pGt2797Nhz/8YcaPH8+UKVNobGxkcHAw6ZodO3bw7ne/mzFjxnD22Wfz2GOPuf14BcnDDz/MO97xjrgz+hkzZvCTn/wk/r20l/9Zvnw5oVCIhoaG+GfSboJgA1UAbNiwQY0ePVo9+uij6g9/+IO67rrr1KRJk9T+/fvzXbSC58c//rH64he/qFpbWxWgtm7dmvT98uXLVXFxsXr88cfV73//e/XRj35UvfWtb1XHjx+PX/OhD31InX/++epXv/qVam9vV2effbZavHhx/Pu+vj5VUVGhrrrqKvXyyy+r9evXq3HjxqlvfOMbXj1mQXHJJZeo73znO+rll19Wu3btUv/1X/+lzjzzTHXkyJH4NTfccIOqrq5W27dvV7/97W/Ve9/7XvW+970v/v3g4KB6+9vfrubMmaNeeOEF9eMf/1iVlZWp2267LX7N3//+dzV+/Hi1dOlS9cc//lE98MADKhKJqCeffNLT5y0EfvCDH6gf/ehH6i9/+Yt65ZVX1O23365OO+009fLLLyulpL38znPPPaemT5+u3vGOd6j6+vr459JugmCdghBcL7zwQnXjjTfG/45Go+qMM85Q99xzTx5LNfJIFVxjsZiaOnWqWrFiRfyzQ4cOqTFjxqj169crpZT64x//qAD1m9/8Jn7NT37yExUKhdSePXuUUko99NBDavLkyWpgYCB+zX//93+rc8891+UnGhl0dXUpQO3cuVMppbXRaaedpjZv3hy/5k9/+pMC1LPPPquU0hYs4XBY7du3L37Nww8/rIqKiuLtdMstt6i3ve1tSXktXLhQXXLJJW4/0ohg8uTJ6pFHHpH28jmHDx9W55xzjnrqqafUzJkz44KrtJsg2CPwpgInTpzg+eefZ86cOfHPwuEwc+bM4dlnn81jyYTXXnuNffv2JbVNcXExF110Ubxtnn32WSZNmsR73vOe+DVz5swhHA7z61//On7Nf/7nfzJ69Oj4NZdccgmvvPIKBw8e9OhpCpe+vj4ASkpKAHj++ec5efJkUrv9y7/8C2eeeWZSu/3bv/0bFRUV8WsuueQS+vv7+cMf/hC/JvEe+jXyXuZGNBplw4YNHD16lBkzZkh7+Zwbb7yRD3/4w8PqVtpNEOwxKt8FyJWenh6i0WjSiw1QUVHBn//85zyVSgDYt28fgGHb6N/t27ePKVOmJH0/atQoSkpKkq5561vfOuwe+neTJ092pfwjgVgsRkNDA+9///t5+9vfDmh1Onr0aCZNmpR0bWq7GbWr/l2ma/r7+zl+/Djjxo1z45EKlpdeeokZM2bw5ptvMmHCBLZu3cp5553Hrl27pL18yoYNG/jd737Hb37zm2HfyXsmCPYIvOAqCIJ9brzxRl5++WV+/vOf57soQhbOPfdcdu3aRV9fH1u2bOHqq69m586d+S6WkIaOjg7q6+t56qmnGDt2bL6LIwgFQ+BNBcrKyohEIsNOYu7fv5+pU6fmqVQCEK//TG0zdepUurq6kr4fHBykt7c36RqjeyTmIVhnyZIl/PCHP6StrY2qqqr451OnTuXEiRMcOnQo6frUdsvWJumuKSoqEi2QDUaPHs3ZZ5/NBRdcwD333MP555/PmjVrpL18yvPPP09XVxfvfve7GTVqFKNGjWLnzp3cf//9jBo1ioqKCmk3QbBB4AXX0aNHc8EFF7B9+/b4Z7FYjO3btzNjxow8lkx461vfytSpU5Papr+/n1//+tfxtpkxYwaHDh3i+eefj1/z9NNPE4vFuOiii+LXPPPMM5w8eTJ+zVNPPcW5554rZgI2UEqxZMkStm7dytNPPz3MDOOCCy7gtNNOS2q3V155hd27dye120svvZS06HjqqacoKirivPPOi1+TeA/9GnkvnSEWizEwMCDt5VMuvvhiXnrpJXbt2hVP73nPe7jqqqvi/y/tJgg2yPfpMCfYsGGDGjNmjHrsscfUH//4R3X99derSZMmJZ3EFNzh8OHD6oUXXlAvvPCCAtTKlSvVCy+8oN544w2llOYOa9KkSeqJJ55QL774opo7d66hO6x3vetd6te//rX6+c9/rs4555wkd1iHDh1SFRUV6uMf/7h6+eWX1YYNG9T48ePFHZZNPvOZz6ji4mK1Y8cOtXfv3ng6duxY/JobbrhBnXnmmerpp59Wv/3tb9WMGTPUjBkz4t/rbno++MEPql27dqknn3xSlZeXG7rpaWxsVH/605/Ugw8+KG56bHLrrbeqnTt3qtdee029+OKL6tZbb1WhUEj97Gc/U0pJewWFRK8CSkm7CYIdCkJwVUqpBx54QJ155plq9OjR6sILL1S/+tWv8l2kEUFbW5sChqWrr75aKaW5xLrjjjtURUWFGjNmjLr44ovVK6+8knSPAwcOqMWLF6sJEyaooqIide2116rDhw8nXfP73/9e/cd//IcaM2aMqqysVMuXL/fqEQsOo/YC1He+8534NcePH1ef/exn1eTJk9X48ePVvHnz1N69e5Pu8/rrr6tLL71UjRs3TpWVlambb75ZnTx5MumatrY29c53vlONHj1a/dM//VNSHoJ5PvnJT6q3vOUtavTo0aq8vFxdfPHFcaFVKWmvoJAquEq7CYJ1QkoplR9dryAIgiAIgiCYJ/A2roIgCIIgCMLIQARXQRAEQRAEIRCI4CoIgiAIgiAEAhFcBUEQBEEQhEAggqsgCIIgCIIQCERwFQRBEARBEAKBCK6CIAiCIAhCIBDBVRAEQRAEQQgEIrgKgiAIgiAIgUAEV0EQBEEQBCEQiOAqCIIgCIIgBAIRXAVBEARBEIRA8P8AT+dEtsgEibgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "x = np.arange(0, len(df), 1)\n",
    "ax.scatter(x, df[\"actual values\"], c='b', label=\"Acutual Values\")\n",
    "ax.scatter(x, df[\"predictions\"], c='r', label=\"Predictions\")\n",
    "ax.legend(loc=(1, 0.5));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mean Squared Error (MSE)**\n",
    "\n",
    "How about MSE? \n",
    "\n",
    "We can calculate it with Scikit-Learn's [`sklearn.metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.2542443610174998)"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean squared error\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "mse = mean_squared_error(y_test, y_preds)\n",
    "mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MSE will often be higher than MAE because is squares the errors rather than only taking the absolute difference into account.\n",
    "\n",
    "Now you might be thinking, which regression evaluation metric should you use?\n",
    "\n",
    "* R^2 is similar to accuracy. It gives you a quick indication of how well your model might be doing. Generally, the closer your R^2 value is to 1.0, the better the model. But it doesn't really tell exactly how wrong your model is in terms of how far off each prediction is.\n",
    "* MAE gives a better indication of how far off each of your model's predictions are on average.\n",
    "* As for MAE or MSE, because of the way MSE is calculated, squaring the differences between predicted values and actual values, it amplifies larger differences. Let's say we're predicting the value of houses (which we are). \n",
    "    * Pay more attention to MAE: When being \\$10,000 off is ***twice*** as bad as being \\$5,000 off.\n",
    "    * Pay more attention to MSE: When being \\$10,000 off is ***more than twice*** as bad as being \\$5,000 off.\n",
    "    \n",
    "> **Note:** What we've covered here is only a handful of potential metrics you can use to evaluate your models. If you're after a complete list, check out the [Scikit-Learn metrics and scoring documentation](https://scikit-learn.org/stable/modules/model_evaluation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.3 Evaluating a model using the `scoring` parameter\n",
    "\n",
    "We've covered a bunch of ways to evaluate a model's predictions but haven't even touched the `scoring` parameter...\n",
    "\n",
    "Not to worry, it's very similar to what we've been doing!\n",
    "\n",
    "As a refresh, the `scoring` parameter can be used with a function like `cross_val_score()` to tell Scikit-Learn what evaluation metric to return using cross-validation.\n",
    "\n",
    "Let's check it out with our classification model and the heart disease dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll use the default, which is mean accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.81967213, 0.90163934, 0.83606557, 0.78333333, 0.78333333])"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_acc = cross_val_score(clf, X, y, cv=5)\n",
    "cv_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen this before, now we got 5 different accuracy scores on different test splits of the data.\n",
    "\n",
    "Averaging this gives the cross-validated accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated accuracy is: 82.48%\n"
     ]
    }
   ],
   "source": [
    "# Cross-validated accuracy\n",
    "print(f\"The cross-validated accuracy is: {np.mean(cv_acc)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find the same using the `scoring` parameter and passing it `\"accuracy\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated accuracy is: 82.48%\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_acc = cross_val_score(clf, X, y, cv=5, scoring=\"accuracy\")\n",
    "print(f\"The cross-validated accuracy is: {np.mean(cv_acc)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same goes for the other metrics we've been using for classification.\n",
    "\n",
    "Let's try `\"precision\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated precision is: 0.83\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_precision = cross_val_score(clf, X, y, cv=5, scoring=\"precision\")\n",
    "print(f\"The cross-validated precision is: {np.mean(cv_precision):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about `\"recall\"`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated recall is: 0.85\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_recall = cross_val_score(clf, X, y, cv=5, scoring=\"recall\")\n",
    "print(f\"The cross-validated recall is: {np.mean(cv_recall):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `\"f1\"` (for F1 score)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated F1 score is: 0.84\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_f1 = cross_val_score(clf, X, y, cv=5, scoring=\"f1\")\n",
    "print(f\"The cross-validated F1 score is: {np.mean(cv_f1):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can repeat this process with our regression metrics.\n",
    "\n",
    "Let's revisit our regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "model = RandomForestRegressor(n_estimators=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default is `\"r2\"`.\n",
    "\n",
    "> **Note:** We can time how long a single cell of code takes to run using the [`%%time` magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated R^2 score is: 0.65\n",
      "CPU times: user 40.5 s, sys: 286 ms, total: 40.8 s\n",
      "Wall time: 41.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "np.random.seed(42)\n",
    "cv_r2 = cross_val_score(model, X, y, cv=5, scoring=\"r2\")\n",
    "print(f\"The cross-validated R^2 score is: {np.mean(cv_r2):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can use `\"neg_mean_absolute_error\"` for MAE (mean absolute error).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated MAE score is: -0.47\n",
      "CPU times: user 40.4 s, sys: 246 ms, total: 40.7 s\n",
      "Wall time: 41.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "np.random.seed(42)\n",
    "cv_mae = cross_val_score(model, X, y, cv=5, scoring=\"neg_mean_absolute_error\")\n",
    "print(f\"The cross-validated MAE score is: {np.mean(cv_mae):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why the `\"neg_\"`?\n",
    "\n",
    "Because Scikit-Learn documentation states:\n",
    "> [\"All scorer objects follow the convention that higher return values are better than lower return values.\"](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)\n",
    "\n",
    "Which in this case, means a lower negative value (closer to 0) is better.\n",
    "\n",
    "What about `\"neg_mean_squared_error\"` for MSE (mean squared error)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross-validated MSE score is: -0.43\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "cv_mse = cross_val_score(model, \n",
    "                         X, \n",
    "                         y, \n",
    "                         cv=5,\n",
    "                         scoring=\"neg_mean_squared_error\")\n",
    "print(f\"The cross-validated MSE score is: {np.mean(cv_mse):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Using different evaluation metrics with Scikit-Learn\n",
    "\n",
    "Remember the third way of evaluating Scikit-Learn functions?\n",
    "\n",
    "> 3. Problem-specific metric functions. Similar to how the `scoring` parameter can be passed different scoring functions, Scikit-Learn implements these as stand alone functions.\n",
    "\n",
    "Well, we've kind of covered this third way of using evaulation metrics with Scikit-Learn.\n",
    "\n",
    "In essence, all of the metrics we've seen previously have their own function in Scikit-Learn.\n",
    "\n",
    "They all work by comparing an array of predictions, usually called `y_preds` to an array of actual labels, usually called `y_test` or `y_true`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification functions\n",
    "For:\n",
    "* Accuracy we can use [`sklearn.metrics.accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)\n",
    "* Precision we can use [`sklearn.metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html)\n",
    "* Recall we can use [`sklearn.metrics.recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html)\n",
    "* F1 we can use [`sklearn.metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier metrics on the test set:\n",
      "Accuracy: 85.25%\n",
      "Precision: 0.85\n",
      "Recall: 0.88\n",
      "F1: 0.86\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=100)\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_preds = clf.predict(X_test)\n",
    "\n",
    "# Evaluate the classifier\n",
    "print(\"Classifier metrics on the test set:\")\n",
    "print(f\"Accuracy: {accuracy_score(y_test, y_preds) * 100:.2f}%\")\n",
    "print(f\"Precision: {precision_score(y_test, y_preds):.2f}\")\n",
    "print(f\"Recall: {recall_score(y_test, y_preds):.2f}\")\n",
    "print(f\"F1: {f1_score(y_test, y_preds):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regression metrics\n",
    "\n",
    "We can use a similar setup for our regression problem, just with different methods.\n",
    "\n",
    "For:\n",
    "* R^2 we can use [`sklearn.metrics.r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html)\n",
    "* MAE (mean absolute error) we can use [`sklearn.metrics.mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html)\n",
    "* MSE (mean squared error) we can use [`sklearn.metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Regression model metrics on the test set:\n",
      "R^2: 0.81\n",
      "MAE: 0.33\n",
      "MSE: 0.25\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X = housing_df.drop(\"target\", axis=1)\n",
    "y = housing_df[\"target\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y, \n",
    "                                                    test_size=0.2)\n",
    "\n",
    "model = RandomForestRegressor(n_estimators=100, \n",
    "                              n_jobs=-1)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_preds = model.predict(X_test)\n",
    "\n",
    "# Evaluate the model\n",
    "print(\"Regression model metrics on the test set:\")\n",
    "print(f\"R^2: {r2_score(y_test, y_preds):.2f}\")\n",
    "print(f\"MAE: {mean_absolute_error(y_test, y_preds):.2f}\")\n",
    "print(f\"MSE: {mean_squared_error(y_test, y_preds):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow!\n",
    "\n",
    "We've covered a lot! \n",
    "\n",
    "But it's worth it. \n",
    "\n",
    "Because evaluating a model's predictions is as important as training a model in any machine learning project.\n",
    "\n",
    "There's nothing worse than training a machine learning model and optimizing for the wrong evaluation metric.\n",
    "\n",
    "Keep the metrics and evaluation methods we've gone through when training your future models.\n",
    "\n",
    "If you're after extra reading, I'd go through the [Scikit-Learn guide for model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html).\n",
    "\n",
    "Now we've seen some different metrics we can use to evaluate a model, let's see some ways we can improve those metrics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Improving model predictions through experimentation (hyperparameter tuning)\n",
    "\n",
    "The first predictions you make with a model are generally referred to as **baseline predictions**. \n",
    "\n",
    "It's similar for the first evaluation metrics you get. These are generally referred to as **baseline metrics**.\n",
    "\n",
    "Your next goal is to improve upon these baseline metrics.\n",
    "\n",
    "How?\n",
    "\n",
    "*Experiment, experiment, experiment!*\n",
    "\n",
    "Two of the main methods to improve baseline metrics are:\n",
    "1. From a data perspective.\n",
    "2. From a model perspective.\n",
    "\n",
    "From a data perspective asks:\n",
    "* Could we collect more data? In machine learning, more data is generally better, as it gives a model more opportunities to learn patterns.\n",
    "* Could we improve our data? This could mean filling in misisng values or finding a better encoding (turning data into numbers) strategy.\n",
    "\n",
    "From a model perspective asks:\n",
    "* Is there a better model we could use? If you've started out with a simple model, could you use a more complex one? (we saw an example of this when looking at the [Scikit-Learn machine learning map](https://scikit-learn.org/stable/machine_learning_map.html), ensemble methods are generally considered more complex models)\n",
    "* Could we improve the current model? If the model you're using performs well straight out of the box, can the hyperparameters be tuned to make it even better?\n",
    "\n",
    "> **Note:** Patterns in data are also often referred to as data parameters. The difference between *parameters* and *hyperparameters* is a machine learning model seeks to find parameters in data on its own, where as, hyperparameters are settings on a model which a person (you) can adjust.\n",
    "\n",
    "Since we have two existing datasets, we'll look at improving our results from a model perspective.\n",
    "\n",
    "More specifically, we'll look at how we could improve our `RandomForestClassifier` and `RandomForestRegressor` models through hyperparameter tuning.\n",
    "\n",
    "What even are hyperparameters?\n",
    "\n",
    "Good question, let's check them out. \n",
    "\n",
    "First, we'll instantiate a `RandomForestClassifier`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we instantiate a model like above, we're using the default hyperparameters.\n",
    "\n",
    "These get printed out when you call the model instance and `get_params()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'class_weight': None,\n",
       " 'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'monotonic_cst': None,\n",
       " 'n_estimators': 100,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see things like `max_depth`, `min_samples_split`, `n_estimators`.\n",
    "\n",
    "Each of these is a hyperparameter of the `RandomForestClassifier` you can adjust. \n",
    "\n",
    "You can think of hyperparameters as being similar to dials on an oven. \n",
    "\n",
    "On the default setting your oven might do an okay job cooking your favourite meal. But with a little experimentation, you find it does better when you adjust the settings. \n",
    "\n",
    "<img src=\"../images/sklearn-hyperparameter-tuning-oven.png\" width=500/>\n",
    "\n",
    "The same goes for imporving a machine learning model by hyperparameter tuning. \n",
    "\n",
    "The default hyperparameters on a machine learning model may find patterns in data well. But there's a chance a adjusting the hyperparameters may improve a models performance.\n",
    "\n",
    "Every machine learning model will have different hyperparameters you can tune.\n",
    "\n",
    "You might be thinking, \"how the hell do I remember all of these?\"\n",
    "\n",
    "Another good question. \n",
    "\n",
    "It's why we're focused on the Random Forest. \n",
    "\n",
    "Instead of memorizing all of the hyperparameters for every model, we'll see how it's done with one. \n",
    "\n",
    "And then knowing these principles, you can apply them to a different model if needed.\n",
    "\n",
    "Reading the [Scikit-Learn documentation for the Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), you'll find they suggest trying to change `n_estimators` (the number of trees in the forest) and `min_samples_split` (the minimum number of samples required to split an internal node).\n",
    "\n",
    "We'll try tuning these as well as:\n",
    "* `max_features` (the number of features to consider when looking for the best split)\n",
    "* `max_depth` (the maximum depth of the tree)\n",
    "* `min_samples_leaf` (the minimum number of samples required to be at a leaf node)\n",
    "\n",
    "If this still sounds like a lot, the good news is, the process we're taking with the Random Forest and tuning its hyperparameters, can be used for other machine learning models in Scikit-Learn. The only difference is, with a different model, the hyperparameters you tune will be different.\n",
    "\n",
    "Adjusting hyperparameters is usually an experimental process to figure out which are best. As there's no real way of knowing which hyperparameters will be best when starting out.\n",
    "\n",
    "To get familar with hyparameter tuning, we'll take our RandomForestClassifier and adjust its hyperparameters in 3 ways.\n",
    "\n",
    "1. By hand\n",
    "2. Randomly with [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "3. Exhaustively with [`sklearn.model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Tuning hyperparameters by hand \n",
    "\n",
    "So far we've worked with training and test datasets.\n",
    "\n",
    "You train a model on a training set and evaluate it on a test dataset.\n",
    "\n",
    "But hyperparameter tuning introduces a thrid set, a validation set.\n",
    "\n",
    "Now the process becomes: \n",
    "1. Train a model on the training data.\n",
    "2. (Try to) improve the model's hyperparameters on the validation set.\n",
    "3. Evaluate the model on the test set.\n",
    "\n",
    "If our starting dataset contained 100 different patient records labels indicating who had heart disease and who didn't and we wanted to build a machine learning model to predict who had heart disease and who didn't, it might look like this:\n",
    "\n",
    "<img src=\"../images/sklearn-train-valid-test-annotated.png\" width=500/>\n",
    "\n",
    "Since we know we're using a `RandomForestClassifier` and we know the hyperparameters we want to adjust, let's see what it looks like.\n",
    "\n",
    "First, let's remind ourselves of the base parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': True,\n",
       " 'ccp_alpha': 0.0,\n",
       " 'class_weight': None,\n",
       " 'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_leaf_nodes': None,\n",
       " 'max_samples': None,\n",
       " 'min_impurity_decrease': 0.0,\n",
       " 'min_samples_leaf': 1,\n",
       " 'min_samples_split': 2,\n",
       " 'min_weight_fraction_leaf': 0.0,\n",
       " 'monotonic_cst': None,\n",
       " 'n_estimators': 100,\n",
       " 'n_jobs': None,\n",
       " 'oob_score': False,\n",
       " 'random_state': None,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we're going to adjust:\n",
    "* `max_depth`\n",
    "* `max_features`\n",
    "* `min_samples_leaf`\n",
    "* `min_samples_split`\n",
    "* `n_estimators`\n",
    "\n",
    "We'll use the same code as before, except this time we'll create a training, validation and test split.\n",
    "\n",
    "With the training set containing 70% of the data and the validation and test sets each containing 15%.\n",
    "\n",
    "Let's get some baseline results, then we'll tune the model.\n",
    "\n",
    "And since we're going to be evaluating a few models, let's make an evaluation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_preds(y_true: np.array, \n",
    "                   y_preds: np.array) -> dict:\n",
    "    \"\"\"\n",
    "    Performs evaluation comparison on y_true labels vs. y_pred labels.\n",
    "\n",
    "    Returns several metrics in the form of a dictionary.\n",
    "    \"\"\"\n",
    "    accuracy = accuracy_score(y_true, y_preds)\n",
    "    precision = precision_score(y_true, y_preds)\n",
    "    recall = recall_score(y_true, y_preds)\n",
    "    f1 = f1_score(y_true, y_preds)\n",
    "    metric_dict = {\"accuracy\": round(accuracy, 2),\n",
    "                   \"precision\": round(precision, 2), \n",
    "                   \"recall\": round(recall, 2),\n",
    "                   \"f1\": round(f1, 2)}\n",
    "    print(f\"Acc: {accuracy * 100:.2f}%\")\n",
    "    print(f\"Precision: {precision:.2f}\")\n",
    "    print(f\"Recall: {recall:.2f}\")\n",
    "    print(f\"F1 score: {f1:.2f}\")\n",
    "\n",
    "    return metric_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful!\n",
    "\n",
    "Now let's recreate a previous workflow, except we'll add in the creation of a validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 80.00%\n",
      "Precision: 0.78\n",
      "Recall: 0.88\n",
      "F1 score: 0.82\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.8,\n",
       " 'precision': np.float64(0.78),\n",
       " 'recall': np.float64(0.88),\n",
       " 'f1': np.float64(0.82)}"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Set the seed\n",
    "np.random.seed(42)\n",
    "\n",
    "# Read in the data\n",
    "# heart_disease = pd.read_csv(\"../data/heart-disease.csv\") # load in from local directory\n",
    "heart_disease = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/heart-disease.csv\") # load directly from URL (source: https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/heart-disease.csv)\n",
    "\n",
    "# Split into X (features) & y (labels)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Training and test split (70% train, 30% test)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)\n",
    "\n",
    "# Create validation and test split by spliting testing data in half (30% test -> 15% validation, 15% test)\n",
    "X_valid, X_test, y_valid, y_test = train_test_split(X_test, y_test, test_size=0.5)\n",
    "\n",
    "clf = RandomForestClassifier()\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_preds = clf.predict(X_valid)\n",
    "\n",
    "# Evaluate the classifier\n",
    "baseline_metrics = evaluate_preds(y_valid, y_preds)\n",
    "baseline_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data: 212 samples, 212 labels\n",
      "Validation data: 45 samples, 45 labels\n",
      "Testing data: 46 samples, 46 labels\n"
     ]
    }
   ],
   "source": [
    "# Check the sizes of the splits\n",
    "print(f\"Training data: {len(X_train)} samples, {len(y_train)} labels\")\n",
    "print(f\"Validation data: {len(X_valid)} samples, {len(y_valid)} labels\")\n",
    "print(f\"Testing data: {len(X_test)} samples, {len(y_test)} labels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful, now let's try and improve the results.\n",
    "\n",
    "We'll change 1 of the hyperparameters, `n_estimators=100` (default) to `n_estimators=200` and see if it improves on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 77.78%\n",
      "Precision: 0.77\n",
      "Recall: 0.83\n",
      "F1 score: 0.80\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "# Create a second classifier\n",
    "clf_2 = RandomForestClassifier(n_estimators=200)\n",
    "clf_2.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "y_preds_2 = clf_2.predict(X_valid)\n",
    "\n",
    "# Evaluate the 2nd classifier\n",
    "clf_2_metrics = evaluate_preds(y_valid, y_preds_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, it looks like doubling the `n_estimators` value performs *worse* than the default, perhaps there's a better value for `n_estimators`?\n",
    "\n",
    "And what other hyperparameters could we change?\n",
    "\n",
    "Wait...\n",
    "\n",
    "This could take a while if all we're doing is building new models with new hyperparameters each time.\n",
    "\n",
    "Surely there's a better way?\n",
    "\n",
    "There is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Hyperparameter tuning with [`RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "\n",
    "Scikit-Learn's [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) allows us to randomly search across different hyperparameters to see which work best. \n",
    "\n",
    "It also stores details about the ones which work best!\n",
    "\n",
    "Let's see it in action.\n",
    "\n",
    "First, we create a dictionary of parameter distributions (collections of different values for specific hyperparamters) we'd like to search over.\n",
    "\n",
    "This dictionary comes in the form:\n",
    "\n",
    "```python\n",
    "param_distributions = {\"hyperparameter_name\": [values_to_randomly_try]}\n",
    "```\n",
    "\n",
    "Where `\"hyperparameter_name\"` is the value of a specific hyperparameter for a model and `[values_to_randomly_try]` is a list of values for that specific hyperparamter to randomly try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameter grid RandomizedSearchCV will search over\n",
    "param_distributions = {\"n_estimators\": [10, 100, 200, 500, 1000, 1200],\n",
    "                       \"max_depth\": [None, 5, 10, 20, 30],\n",
    "                       \"max_features\": [\"sqrt\", \"log2\", None],\n",
    "                       \"min_samples_split\": [2, 4, 6, 8],\n",
    "                       \"min_samples_leaf\": [1, 2, 4, 8]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where did these values come from?\n",
    "\n",
    "They're made up.\n",
    "\n",
    "Made up?\n",
    "\n",
    "Yes. \n",
    "\n",
    "Not completely pulled out of the air but after reading the [Scikit-Learn documentation on Random Forest's](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) you'll see some of these values have certain values which usually perform well and certain hyperparameters take strings rather than integers.\n",
    "\n",
    "Now we've got the parameter distribution dictionary setup, Scikit-Learn's `RandomizedSearchCV` will look at it, pick a random value from each, instantiate a model with those values and test each model.\n",
    "\n",
    "How many models will it test?\n",
    "\n",
    "As many as there are for each combination of hyperparameters to be tested. Let's add them up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1440 potential combinations of hyperparameters to test.\n"
     ]
    }
   ],
   "source": [
    "# Count the total number of hyperparameter combinations to test\n",
    "total_randomized_hyperparameter_combintions_to_test = np.prod([len(value) for value in param_distributions.values()])\n",
    "print(f\"There are {total_randomized_hyperparameter_combintions_to_test} potential combinations of hyperparameters to test.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah!\n",
    "\n",
    "That's a lot of combinations!\n",
    "\n",
    "Or...\n",
    "\n",
    "We can set the `n_iter` parameter to limit the number of models `RandomizedSearchCV` tests (e.g. `n_iter=20` means to try `20` different random combintations of hyperparameters and will cross-validate each set, so if `cv=5`, 5x20 = 100 total fits).\n",
    "\n",
    "The best thing? \n",
    "\n",
    "The results we get will be cross-validated (hence the CV in `RandomizedSearchCV`) so we can use `train_test_split()`.\n",
    "\n",
    "And since we're going over so many different models, we'll set `n_jobs=-1` in our [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) so Scikit-Learn takes advantage of all the cores (processors) on our computers.\n",
    "\n",
    "Let's see it in action.\n",
    "\n",
    "> **Note:** Depending on `n_iter` (how many models you test), the different values in the hyperparameter grid, and the power of your computer, running the cell below may take a while (for reference, it took about ~1-minute on my M1 Pro MacBook Pro)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 30 candidates, totalling 150 fits\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=None, min_samples_leaf=8, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=10, max_features=None, min_samples_leaf=8, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=10, max_features=None, min_samples_leaf=8, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=10, max_features=None, min_samples_leaf=8, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=10, max_features=None, min_samples_leaf=8, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=2, min_samples_split=8, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=4, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=2, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=8, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=8, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=8, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=8, min_samples_split=4, n_estimators=1200; total time=   0.8s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=8, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=8, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=log2, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=1200; total time=   0.8s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=1200; total time=   0.8s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=4, min_samples_split=4, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=5, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=5, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1200; total time=   0.8s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1200; total time=   0.8s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=2, min_samples_split=8, n_estimators=1200; total time=   0.7s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=8, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=8, min_samples_split=2, n_estimators=1000; total time=   0.7s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=8, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=8, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=8, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=2, min_samples_split=6, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=4, min_samples_split=6, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=4, min_samples_split=8, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=5, max_features=None, min_samples_leaf=1, min_samples_split=6, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=sqrt, min_samples_leaf=1, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=log2, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=20, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=2, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=1, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=10, max_features=sqrt, min_samples_leaf=1, min_samples_split=4, n_estimators=100; total time=   0.1s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=500; total time=   0.3s\n",
      "[CV] END max_depth=20, max_features=sqrt, min_samples_leaf=2, min_samples_split=2, n_estimators=500; total time=   0.4s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=None, min_samples_leaf=8, min_samples_split=4, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[CV] END max_depth=None, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=10; total time=   0.0s\n",
      "[INFO] Total time taken for 30 random combinations of hyperparameters: 36.02 seconds.\n"
     ]
    }
   ],
   "source": [
    "# Start the timer\n",
    "import time\n",
    "start_time = time.time()\n",
    "\n",
    "from sklearn.model_selection import RandomizedSearchCV, train_test_split\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split into X & y\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Set n_jobs to -1 to use all available cores on your machine (if this causes errors, try n_jobs=1)\n",
    "clf = RandomForestClassifier(n_jobs=-1)\n",
    "\n",
    "# Setup RandomizedSearchCV \n",
    "n_iter = 30 # try 30 models total\n",
    "rs_clf = RandomizedSearchCV(estimator=clf,\n",
    "                            param_distributions=param_distributions,\n",
    "                            n_iter=n_iter, \n",
    "                            cv=5, # 5-fold cross-validation\n",
    "                            verbose=2) # print out results\n",
    "\n",
    "# Fit the RandomizedSearchCV version of clf (does cross-validation for us, so no need to use a validation set)\n",
    "rs_clf.fit(X_train, y_train);\n",
    "\n",
    "# Finish the timer\n",
    "end_time = time.time()\n",
    "print(f\"[INFO] Total time taken for {n_iter} random combinations of hyperparameters: {end_time - start_time:.2f} seconds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When `RandomizedSearchCV` goes through `n_iter` combinations of of hyperparameter search space, it stores the best ones in the attribute `best_params_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 200,\n",
       " 'min_samples_split': 6,\n",
       " 'min_samples_leaf': 4,\n",
       " 'max_features': 'log2',\n",
       " 'max_depth': 30}"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the best hyperparameters found by RandomizedSearchCV\n",
    "rs_clf.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when we call `predict()` on `rs_clf` (our `RandomizedSearchCV` version of our classifier), it'll use the best hyperparameters it found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 85.25%\n",
      "Precision: 0.85\n",
      "Recall: 0.88\n",
      "F1 score: 0.86\n"
     ]
    }
   ],
   "source": [
    "# Make predictions with the best hyperparameters\n",
    "rs_y_preds = rs_clf.predict(X_test)\n",
    "\n",
    "# Evaluate the predictions\n",
    "rs_metrics = evaluate_preds(y_test, rs_y_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent! \n",
    "\n",
    "Thanks to `RandomizedSearchCV` testing out a bunch of different hyperparameters, we get a nice boost to all of the evaluation metrics for our classification model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Hyperparameter tuning with [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)\n",
    "\n",
    "There's one more way we could try to improve our model's hyperparamters. \n",
    "\n",
    "And it's with [`sklearn.model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html).\n",
    "\n",
    "The main difference between `GridSearchCV` and `RandomizedSearchCV` is `GridSearchCV` searches across a grid of hyperparamters exhaustively (it will try every combination possible), where as, `RandomizedSearchCV` searches across a grid of hyperparameters randomly (stopping after `n_iter` combinations).\n",
    "\n",
    "`GridSearchCV` also refers to a dictionary of parameter distributions as a parameter grid (via the parameter `param_grid`).\n",
    "\n",
    "For example, let's see our dictionary of hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': [10, 100, 200, 500, 1000, 1200],\n",
       " 'max_depth': [None, 5, 10, 20, 30],\n",
       " 'max_features': ['sqrt', 'log2', None],\n",
       " 'min_samples_split': [2, 4, 6, 8],\n",
       " 'min_samples_leaf': [1, 2, 4, 8]}"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`RandomizedSearchCV` tries `n_iter` combinations of different values. \n",
    "\n",
    "Where as, `GridSearchCV` will try every single possible combination. \n",
    "\n",
    "And if you remember from before when we did the calculation: `max_depth` has 4 values, `max_features` has 2, `min_samples_leaf` has 3, `min_samples_split` has 3, `n_estimators` has 5. \n",
    "\n",
    "That's 4x2x3x3x5 = 360 models!\n",
    "\n",
    "This could take a long time depending on the power of the computer you're using, the amount of data you have and the complexity of the hyperparamters (usually higher values means a more complex model).\n",
    "\n",
    "In our case, the data we're using is relatively small (only ~300 samples).\n",
    "\n",
    "Since we've already tried to find some ideal hyperparameters using `RandomizedSearchCV`, we'll create another hyperparameter grid based on the `best_params_` of `rs_clf` with less options and then try to use `GridSearchCV` to find a more ideal set.\n",
    "\n",
    "In essence, the workflow could be:\n",
    "1. Tune hyperparameters by hand to get a feel of the data/model.\n",
    "2. Create a large set of hyperparameter distributions and search across them randomly with `RandomizedSearchCV`.\n",
    "3. Find the best hyperparameters from 2 and reduce the search space before searching across a smaller subset exhaustively with `GridSearchCV`.\n",
    "\n",
    "> **Note:** Based on the `best_params_` of `rs_clf` implies the next set of hyperparameters we'll try are roughly in the same range of the best set found by `RandomizedSearchCV`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create hyperparameter grid similar to rs_clf.best_params_\n",
    "param_grid = {\"n_estimators\": [200, 1000],\n",
    "              \"max_depth\": [30, 40, 50],\n",
    "              \"max_features\": [\"log2\"],\n",
    "              \"min_samples_split\": [2, 4, 6, 8],\n",
    "              \"min_samples_leaf\": [4]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've created another grid of hyperparameters to search over, this time with less total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 24 combinations of hyperparameters to test.\n",
      "This is 60.0 times less than before (previous: 1440).\n"
     ]
    }
   ],
   "source": [
    "# Count the total number of hyperparameter combinations to test\n",
    "total_grid_search_hyperparameter_combinations_to_test = np.prod([len(value) for value in param_grid.values()])\n",
    "print(f\"There are {total_grid_search_hyperparameter_combinations_to_test} combinations of hyperparameters to test.\")\n",
    "print(f\"This is {total_randomized_hyperparameter_combintions_to_test/total_grid_search_hyperparameter_combinations_to_test} times less\\\n",
    " than before (previous: {total_randomized_hyperparameter_combintions_to_test}).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when we run `GridSearchCV`, passing it our classifier (`clf`), parameter grid (`param_grid`) and the number of cross-validation folds we'd like to use (`cv=5`), it'll create a model with every single combination of hyperparameters, and then cross-validate each 5 times (for example, 36 hyperparameter combinations * 5 = 135 fits in total) and check the results.\n",
    "\n",
    "> **Note:** Depending on the compute power of the machine you're using, the following cell may take a few minutes to run (for reference, it took ~60 seconds on my M1 Pro MacBook Pro)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 24 candidates, totalling 120 fits\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=30, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.7s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.7s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=40, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=2, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=4, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=200; total time=   0.2s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=6, n_estimators=1000; total time=   0.7s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=200; total time=   0.1s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.6s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.5s\n",
      "[CV] END max_depth=50, max_features=log2, min_samples_leaf=4, min_samples_split=8, n_estimators=1000; total time=   0.5s\n"
     ]
    }
   ],
   "source": [
    "# Start the timer\n",
    "import time\n",
    "start_time = time.time()\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV, train_test_split\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "# Split into X & y\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "y = heart_disease[\"target\"]\n",
    "\n",
    "# Split into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Set n_jobs to -1 to use all available machine cores (if this produces errors, try n_jobs=1)\n",
    "clf = RandomForestClassifier(n_jobs=-1)\n",
    "\n",
    "# Setup GridSearchCV\n",
    "gs_clf = GridSearchCV(estimator=clf,\n",
    "                      param_grid=param_grid,\n",
    "                      cv=5, # 5-fold cross-validation\n",
    "                      verbose=2) # print out progress\n",
    "\n",
    "# Fit the RandomizedSearchCV version of clf\n",
    "gs_clf.fit(X_train, y_train);\n",
    "\n",
    "# Find the running time\n",
    "end_time = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] The total running time for running GridSearchCV was 41.95 seconds.\n"
     ]
    }
   ],
   "source": [
    "# How long did it take? \n",
    "total_time = end_time - start_time\n",
    "print(f\"[INFO] The total running time for running GridSearchCV was {total_time:.2f} seconds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it completes, we can check the best hyperparameter combinations it found using the `best_params_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 30,\n",
       " 'max_features': 'log2',\n",
       " 'min_samples_leaf': 4,\n",
       " 'min_samples_split': 2,\n",
       " 'n_estimators': 200}"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the best hyperparameters found with GridSearchCV\n",
    "gs_clf.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And by default when we call the `predict()` function on `gs_clf`, it'll use the best hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 88.52%\n",
      "Precision: 0.88\n",
      "Recall: 0.91\n",
      "F1 score: 0.89\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.89,\n",
       " 'precision': np.float64(0.88),\n",
       " 'recall': np.float64(0.91),\n",
       " 'f1': np.float64(0.89)}"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Max predictions with the GridSearchCV classifier\n",
    "gs_y_preds = gs_clf.predict(X_test)\n",
    "\n",
    "# Evaluate the predictions\n",
    "gs_metrics = evaluate_preds(y_test, gs_y_preds)\n",
    "gs_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a DataFrame to compare the different metrics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzoAAALECAYAAADAXkVpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEFUlEQVR4nO3deZyVBd3///cw7LvKqoKAoqKCC9wYaVlIYhpptpgbhKSpkAvkQim4lKApmuktX3dvyyTX/H41N5RMXFBUct8QoQRcSFCIRWZ+f/Rr7iYGZJThwOXz+Xicx4O5zrV8js1JXl7XuU5ZZWVlZQAAAAqkXqkHAAAAWNeEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwqlf6gHWRkVFRd5+++20aNEiZWVlpR4HAAAokcrKynz44YfZfPPNU6/e6s/bbBSh8/bbb6dTp06lHgMAANhAzJkzJ1tuueVqn98oQqdFixZJ/vliWrZsWeJpAACAUlm0aFE6depU1Qirs1GEzr8uV2vZsqXQAQAAPvEjLW5GAAAAFI7QAQAACkfoAAAAhbNRfEZnbVRUVGT58uWlHoMCatCgQcrLy0s9BgAAtVCI0Fm+fHnefPPNVFRUlHoUCqp169bp0KGD73ECANhIbPShU1lZmblz56a8vDydOnVa45cGQW1VVlZmyZIleeedd5IkHTt2LPFEAACsjY0+dD7++OMsWbIkm2++eZo2bVrqcSigJk2aJEneeeedtGvXzmVsAAAbgY3+9MfKlSuTJA0bNizxJBTZvyJ6xYoVJZ4EAIC1sdGHzr/47AR1ye8XAMDGpTChAwAA8C9Cp0S+8pWv5MQTTyzZ8X/wgx/kwAMP3GDmAQCAdWmjvxnB6nQ57a71erxZ4/dfr8db12677bY0aNCg1GMAAMA6UdjQoXY23XTTUo8AAADrjEvXSujjjz/OiBEj0qpVq7Rp0yZnnHFGKisrkyQ33HBD+vTpkxYtWqRDhw459NBDq77LJUn+/ve/57DDDkvbtm3TpEmTdO/ePddee23V83PmzMn3vve9tG7dOptuumkOOOCAzJo1a7Wz/Oela126dMm5556bI488Mi1atEjnzp1zxRVXVNumtscAAID1ReiU0PXXX5/69etn2rRp+dWvfpUJEybkqquuSvLP2xifc845mTFjRu64447MmjUrP/jBD6q2PeOMM/Liiy/mj3/8Y1566aVcfvnladOmTdW2AwcOTIsWLfLnP/85U6dOTfPmzbPvvvtm+fLlaz3fhRdemD59+uSZZ57Jcccdl2OPPTavvPLKOj0GAADUBZeulVCnTp1y0UUXpaysLNttt12ee+65XHTRRTnqqKNy5JFHVq3XrVu3XHLJJfmv//qvfPTRR2nevHlmz56dXXfdNX369EnyzzMw/zJp0qRUVFTkqquuqrot8rXXXpvWrVtnypQp2WeffdZqvv322y/HHXdckuTUU0/NRRddlIceeijbbbfdOjsGAADUBWd0SugLX/hCte9n6devX1577bWsXLky06dPz6BBg9K5c+e0aNEie+21V5Jk9uzZSZJjjz02N910U3bZZZeccsopefTRR6v2M2PGjLz++utp0aJFmjdvnubNm2fTTTfN0qVL88Ybb6z1fL169ar6c1lZWTp06FB1+dy6OgYAANQFZ3Q2QEuXLs3AgQMzcODA/Pa3v03btm0ze/bsDBw4sOqysK9//et56623cvfdd+f+++/P3nvvneHDh+eCCy7IRx99lN69e+e3v/3tKvtu27btWs/xn3dhKysrS0VFRZKss2MAAEBdEDol9MQTT1T7+fHHH0/37t3z8ssv5/3338/48ePTqVOnJMlTTz21yvZt27bNkCFDMmTIkHzpS1/KySefnAsuuCC77bZbJk2alHbt2qVly5Z1Mvv6OAYAAHxaLl0rodmzZ2fkyJF55ZVX8rvf/S6//vWvc8IJJ6Rz585p2LBhfv3rX2fmzJm58847c84551TbdsyYMfnDH/6Q119/PS+88EL+3//7f+nRo0eS5LDDDkubNm1ywAEH5M9//nPefPPNTJkyJccff3z++te/rpPZ18cxAADg0xI6JTR48OD84x//SN++fTN8+PCccMIJOfroo9O2bdtcd911ufnmm7PDDjtk/PjxueCCC6pt27Bhw4wePTq9evXKl7/85ZSXl+emm25KkjRt2jQPP/xwOnfunIMOOig9evTIsGHDsnTp0nV29mV9HAMAAD6tssp/fXHLBmzRokVp1apVFi5cuMpfopcuXZo333wzXbt2TePGjUs0IUXn9wwAYMOwpjb4d87oAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOL4wFADYYLy0fY86P0aPl1+q82MApeeMDgAAUDhCBwAAKByhAwAAFI7QAQAACqe4NyM4s9V6Pt7CdbarWbNmpWvXrnnmmWeyyy67JEmmTp2aY445Ji+//HL233//3HHHHevseAAAUDTO6GwkRo4cmV122SVvvvlmrrvuujWuO2PGjBxyyCHp1KlTmjRpkh49euRXv/rV+hkUAAA2AMU9o1Mwb7zxRo455phsueWWn7ju9OnT065du/zmN79Jp06d8uijj+boo49OeXl5RowYsR6mBQCA0nJGp4QqKipy/vnnZ5tttkmjRo3SuXPn/OIXv6i2zqxZs1JWVpb3338/Rx55ZMrKyj7xjM6RRx6ZX/3qV9lrr73SrVu3HH744Rk6dGhuu+22Onw1AACw4XBGp4RGjx6dK6+8MhdddFH23HPPzJ07Ny+//HK1dTp16pS5c+dmu+22y9lnn52DDz44rVrV/vNHCxcuzKabbrquRgcAgA2a0CmRDz/8ML/61a9y6aWXZsiQIUmSrbfeOnvuuWdmzZpVtV55eXk6dOiQsrKytGrVKh06dKj1sR599NFMmjQpd91117oaHwAANmguXSuRl156KcuWLcvee+9dp8d5/vnnc8ABB2Ts2LHZZ5996vRYAACwoRA6JdKkSZM6P8aLL76YvffeO0cffXROP/30Oj8eAABsKFy6ViLdu3dPkyZNMnny5Pzwhz9c5/t/4YUX0r9//wwZMmSVGxxsqP7x/PN1uv8mO+1Up/sHAFgXXtq+R53uv8fLL9Xp/jcUQqdEGjdunFNPPTWnnHJKGjZsmD322CPvvvtuXnjhhc98Odvzzz+f/v37Z+DAgRk5cmTmzZuX5J+f92nbtu26GB8AADZoxQ2dMxeWeoJPdMYZZ6R+/foZM2ZM3n777XTs2DHHHHPMZ97vLbfcknfffTe/+c1v8pvf/KZq+VZbbVXtRgcAAFBUZZWVlZWlHuKTLFq0KK1atcrChQvTsmXLas8tXbo0b775Zrp27ZrGjRuXaELWhQ350jW/ZwDrR11fspN8fi7bYePl0rU1W1Mb/Ds3IwAAAApH6GyEjjnmmDRv3rzGx7q49A0AADZ2xf2MToGdffbZ+clPflLjc2s6fQds+FyuUHxdTqvbL2+eNX7/Ot1/z+t71un+f1+newc+T4TORqhdu3Zp165dqccAAIANlkvXAACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAonMLeda2ub3/5n54b8tx6PV5t/eAHP8gHH3yQO+64o9SjbBDKyspy++2358ADDyz1KAAA1AFndAAAgMIROhuI5cuXl3qEwlixYkWpRwAAoMSETol85StfyYgRI3LiiSemTZs2GThwYJJkwoQJ6dmzZ5o1a5ZOnTrluOOOy0cffVS13XXXXZfWrVvn3nvvTY8ePdK8efPsu+++mTt3btU6K1euzMiRI9O6detsttlmOeWUU1JZWVnt+MuWLcvxxx+fdu3apXHjxtlzzz3z5JNPVj0/ZcqUlJWV5d57782uu+6aJk2apH///nnnnXfyxz/+MT169EjLli1z6KGHZsmSJat9nW+99VYGDRqUTTbZJM2aNcuOO+6Yu+++u+r5559/Pl//+tfTvHnzdNlrrwwbPTrv/f3vVc/f98gj2Xvw4HT84hez5Z575qDhwzNzzpz/3f/f/pamPXvmlnvuyT4/+EE26d07N931z28dv/7229P7wAPTerfd0vWrX82IESOqzfbee+/lW9/6Vpo2bZru3bvnzjvvXKv/7QAA2PAJnRK6/vrr07Bhw0ydOjUTJ05MktSrVy+XXHJJXnjhhVx//fV58MEHc8opp1TbbsmSJbngggtyww035OGHH87s2bPzk5/8pOr5Cy+8MNddd12uueaaPPLII1mwYEFuv/32avs45ZRTcuutt+b666/P008/nW222SYDBw7MggULqq135pln5tJLL82jjz6aOXPm5Hvf+14uvvji3Hjjjbnrrrty33335de//vVqX+Pw4cOzbNmyPPzww3nuuedy3nnnpXnz5kmSDz74IP3798+uu+6ap556KndMnJh33n8/R/zba1nyj3/kx4MH55GbbspdV12VevXq5fsnnJCKiopqxznj4otz3OGH55k//CED9tgjV0yalJN+8Ysc+Z3v5MnbbsvNv/51ttlmm2rbnHXWWfne976Xv/zlL9lvv/1y2GGHrfL6AQDYOBX2ZgQbg+7du+f888+vtuzEE0+s+nOXLl3y85//PMccc0z++7//u2r5ihUrMnHixGy99dZJkhEjRuTss8+uev7iiy/O6NGjc9BBByVJJk6cmHvvvbfq+cWLF+fyyy/Pddddl69//etJkiuvvDL3339/rr766px88slV6/785z/PHnvskSQZNmxYRo8enTfeeCPdunVLknznO9/JQw89lFNPPbXG1zh79ux8+9vfTs+e/7w5xL+2S5JLL700u+66a84999wkyT8+/jiXn312tv3a1/LarFnp3qVLDvza16rtb+LZZ6fzl7+cl954Izt27161fMThh+fAAQOqfj7viity/ODBGX744VXLvvT971fb1w9+8IMccsghSZJzzz03l1xySaZNm5Z99923xtcCAMDGQ+iUUO/evVdZ9sADD2TcuHF5+eWXs2jRonz88cdZunRplixZkqZNmyZJmjZtWhU5SdKxY8e88847SZKFCxdm7ty52X333auer1+/fvr06VN1+dobb7yRFStWVAVMkjRo0CB9+/bNSy+9VG2eXr16Vf25ffv2adq0abVYad++faZNm7ba13j88cfn2GOPzX333ZcBAwbk29/+dtU+Z8yYkYceeqjqDE/+7SzNzDlz0r1Ll7z+1ls557LL8uRf/pL3P/ig6kzOnLlzq4XObjvuWPXnd95/P3PfeSdf/cIXVjvXf762Zs2apWXLllX/HAEAVqeu7+77+zrd++eHS9dKqFmzZtV+njVrVr7xjW+kV69eufXWWzN9+vRcdtllSarfrKBBgwbVtisrK1vlMzjryr8fq6ysrMZj/+dlZP/uhz/8YWbOnJkjjjgizz33XPr06VN1qdtHH32UQYMG5dlnn82zzz6bx2+5JY/fckueu+uu7Pn/R+B3RozI3xcuzGVnnpk/3Xhj/nTjjUmS5f9xw4GmTZpU/blJ48a1fm1r81oAANh4CJ0NyPTp01NRUZELL7wwX/jCF7Ltttvm7bffrtU+WrVqlY4dO+aJJ56oWvbxxx9n+vTpVT9vvfXWVZ8N+pcVK1bkySefzA477PDZX8h/6NSpU4455pjcdtttGTVqVK688sokyW677ZYXXnghXbp0yTbbbJOtO3euejRr2jTvf/BBXp01K6cefXS++oUvZPtu3fLBokWfeLwWzZplqy22yEOPP77OXwsAABsHobMB2WabbbJixYr8+te/zsyZM3PDDTdU3aSgNk444YSMHz8+d9xxR15++eUcd9xx+eCDD6qeb9asWY499ticfPLJueeee/Liiy/mqKOOypIlSzJs2LB1+Ir++Zmje++9N2+++WaefvrpPPTQQ+nRo0eSf96oYMGCBTnkkEPy5JNPZuacObl/6tQcffrpWblyZTZp2TKbtW6da265JW/Mnp0pTzyRU3/5y7U67s+OPTaX/M//5L9/+9u8/tZbeebFF9d40wQAAIqlsJ/ReW7Ic6UeodZ23nnnTJgwIeedd15Gjx6dL3/5yxk3blwGDx5cq/2MGjUqc+fOzZAhQ1KvXr0ceeSR+da3vpWFCxdWrTN+/PhUVFTkiCOOyIcffpg+ffrk3nvvzSabbLJOX9PKlSszfPjw/PWvf03Lli2z77775qKLLkqSbL755pk6dWpOPfXU7LPPPlm2dGk6d+yYr+2xR+rVq5eysrJcf/75+cn48enzrW9l2y5dcsFpp2XgkUd+4nEPP+CALF22LJfecENGX3BBNttkk3z3P25GAABAcZVV1tWHO9ahRYsWpVWrVlm4cGFatmxZ7bmlS5fmzTffTNeuXdN4LT+bwYbpH88/X6f7b7LTTp96W79nrC8vbd+jTvff4+WXPnkl6lSX0+6q0/3PGr9/ne6/zj+EPe7jOt1/4n3AZ7exvw829vfAmtrg3xX2jA7r3gvvvVCn++/2yatASdX1v9gSd9oBgHXFZ3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6GxkZs2albKysjz77LOrXWfKlCkpKyvLBx98sN7mWheO/tnP8r3jjy/1GAAAFEBhvzC0rr9d/D+tr2+Y7dSpU+bOnZs2bdqsl+MBAMDGyBmdjcjy5ctTXl6eDh06pH79jatRVyxfUeoRAAD4HBE6JfLhhx/msMMOS7NmzdKxY8dcdNFF+cpXvpITTzyxap0uXbrknHPOyeDBg9OyZcscffTRNV66dvfdd2fbbbdNkyZN8tWvfjWzZs1a47ErKytz5plnpnPnzmnUqFE233zzHP9vl4wtW7YsP/nJT7LFFlukWbNm2X333TNlypSq5z9Y8EFOPvrk9O/ZP30698m3vvyt3H3b3dWO8YMDfpBfnPqLjP/Z+Oy53Z45+ntHJ0lef/n1HHfocdm96+7p26VvBn9jcGa/Obvathdfd126fvWr2XLPPXPiz3+eFStEEgAAtbNxnRYokJEjR2bq1Km588470759+4wZMyZPP/10dtlll2rrXXDBBRkzZkzGjh1b437mzJmTgw46KMOHD8/RRx+dp556KqNGjVrjsW+99dZcdNFFuemmm7Ljjjtm3rx5mTFjRtXzI0aMyIsvvpibbropm2++eW6//fbsu+++ue1Pt2WrrbfKsmXLssPOO2TYj4elWYtmefj+hzP6uNHp1KVTeu7Ws2o/f5j0hxw89ODccNcNSZL5c+dnyDeH5L+++F+5+var07xF8zzzxDNZuXJl1TYPP/lkOrRtm3uuvjpvzJmTwSefnF7bb58jv/Od2v4jBgDgc0zolMCHH36Y66+/PjfeeGP23nvvJMm1116bzTfffJV1+/fvXy1c/vNszeWXX56tt946F154YZJku+22y3PPPZfzzjtvtcefPXt2OnTokAEDBqRBgwbp3Llz+vbtW/Xctddem9mzZ1fN85Of/CT33HNPbv/d7Tnx9BPTvmP7DB0+tGp/hx11WKY+NDX3/OGeaqGzVbetMmrs/85+8c8vTouWLfLLK3+ZBg0aJEm6bN2l2mytW7bMRT/9acrLy7Ndt27Z90tfypQnnhA6AGvrzFZ1u/+unet2/wDriNApgZkzZ2bFihVVcZEkrVq1ynbbbbfKun369Fnjvl566aXsvvvu1Zb169dvjdt897vfzcUXX5xu3bpl3333zX777ZdBgwalfv36ee6557Jy5cpsu+221bZZtmxZ6jf/56/LypUrc+XFV+beP9yb+XPnZ8XyFVmxfEWaNGlSbZsddt6h2s+vPP9Kdtt9t6rIqUmPrbdOeXl51c8d2rbNC6+9tsbXAwAA/0nobOCaNWu2zvfZqVOnvPLKK3nggQdy//3357jjjssvf/nL/OlPf8pHH32U8vLyTJ8+vVpwJMnflv0tSXLtpdfmN1f8Jqf+/NR079E9TZs2zfjTx6/yWZomTauHT6PGjT5xtgb/cZOFsrKyVFRUfJqXCQDA55ibEZRAt27d0qBBgzz55JNVyxYuXJhXX3211vvq0aNHpk2bVm3Z448//onbNWnSJIMGDcoll1ySKVOm5LHHHstzzz2XXXfdNStXrsw777yTbbbZptqjTft/3tL6mWnP5Kv7fjWDvjso2++0fbbssmXeeuOtTzzmtjtum6efeNrNBQAAqHNCpwRatGiRIUOG5OSTT85DDz2UF154IcOGDUu9evVSVlZWq30dc8wxee2113LyySfnlVdeyY033pjrrrtujdtcd911ufrqq/P8889n5syZ+c1vfpMmTZpkq622yrbbbpvDDjssgwcPzm233ZY333wz06ZNy7hx4/Kn+/6U5J+fvXnsT4/lmWnP5I1X38hZo87K++++/4mzHjrs0Hz04Uc5+aiT8/yzz+etN97Knb+/M2++/matXjMAAHwSoVMiEyZMSL9+/fKNb3wjAwYMyB577JEePXqkcePGtdpP586dc+utt+aOO+7IzjvvnIkTJ+bcc89d4zatW7fOlVdemT322CO9evXKAw88kP/7f/9vNttssyT/vDHC4MGDM2rUqGy33XY58MAD8+STT6bjlh2TJD8a+aPs0GuH/Oh7P8rQA4emTbs26f/1/p84a+tNW+fq267OksVLMvSAofnegO/l1htu3ei+EwgAgA1fWWVlZWWph/gkixYtSqtWrbJw4cK0bNmy2nNLly7Nm2++ma5du9Y6EjYkixcvzhZbbJELL7www4YNK/U4NXrhvRfqdP/d5tXtr2KTnXb61NsW5feMz6bn9T0/eaXP6PfjPq7T/fd4+aU63T+frMtpd9Xp/mc1PrRO99+zju+6VtfvgcT7gM+urv994N8Fa7amNvh3/lN6iTzzzDN5+eWX07dv3yxcuDBnn312kuSAAw4o8WQAAHWnrmM/SWaN37/Oj8GGT+iU0AUXXJBXXnklDRs2TO/evfPnP/85bdq0KfVYAACw0RM6JbLrrrtm+vTppR4DAAAKyc0IAACAwhE6AABA4RQmdDaCm8exEauoqCj1CAAA1MJG/xmdBg0apKysLO+++27atm1b6y/cZO1VrKjbv+wvq6jbWC1burTW21RWVmb58uV59913U69evTRs2LAOJgMAYF3b6EOnvLw8W265Zf76179m1qxZpR6n0N756J063X/lojrdfRo0aPCpt23atGk6d+6cevUKcxIUAKDQNvrQSZLmzZune/fuWbFiRalHKbQTbj+hTvd/0RV1++VYXf9496farry8PPXr13e2EABgI/KpQueyyy7LL3/5y8ybNy8777xzfv3rX6dv376rXf/iiy/O5ZdfntmzZ6dNmzb5zne+k3Hjxq3Tb5gvLy9PeXn5Otsfq5q7fG6d7r/e3LoNnXX5+wYAwIat1tfhTJo0KSNHjszYsWPz9NNPZ+edd87AgQPzzjs1X9Z044035rTTTsvYsWPz0ksv5eqrr86kSZPy05/+9DMPDwAAUJNah86ECRNy1FFHZejQodlhhx0yceLENG3aNNdcc02N6z/66KPZY489cuihh6ZLly7ZZ599csghh2TatGmfeXgAAICa1Cp0li9fnunTp2fAgAH/u4N69TJgwIA89thjNW7zxS9+MdOnT68Km5kzZ+buu+/Ofvvtt9rjLFu2LIsWLar2AAAAWFu1+ozOe++9l5UrV6Z9+/bVlrdv3z4vv/xyjdsceuihee+997LnnnumsrIyH3/8cY455pg1Xro2bty4nHXWWbUZbYPX5bS76nT/s8bvX6f7BwDYaJzZqm7337Vz3e6fdaLO75U7ZcqUnHvuufnv//7vPP3007ntttty11135ZxzzlntNqNHj87ChQurHnPmzKnrMQEAgAKp1RmdNm3apLy8PPPnz6+2fP78+enQoUON25xxxhk54ogj8sMf/jBJ0rNnzyxevDhHH310fvazn9X4vSSNGjVKo0aNajMaAABAlVqd0WnYsGF69+6dyZMnVy2rqKjI5MmT069fvxq3WbJkySox86/bQFdWVtZ2XgAAgE9U6+/RGTlyZIYMGZI+ffqkb9++ufjii7N48eIMHTo0STJ48OBsscUWGTduXJJk0KBBmTBhQnbdddfsvvvuef3113PGGWdk0KBBvvcGAACoE7UOnYMPPjjvvvtuxowZk3nz5mWXXXbJPffcU3WDgtmzZ1c7g3P66aenrKwsp59+ev72t7+lbdu2GTRoUH7xi1+su1cBAADwb2odOkkyYsSIjBgxosbnpkyZUv0A9etn7NixGTt27Kc5FAAAQK3V+V3XAAAA1jehAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4Xyq20uzATqzVd0fo2vnuj8GhdbltLvqdP+zxu9fp/sHADYezugAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA47roGFEdd333QnQcBYKPhjA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDifKnQuu+yydOnSJY0bN87uu++eadOmrXH9Dz74IMOHD0/Hjh3TqFGjbLvttrn77rs/1cAAAACfpH5tN5g0aVJGjhyZiRMnZvfdd8/FF1+cgQMH5pVXXkm7du1WWX/58uX52te+lnbt2uWWW27JFltskbfeeiutW7deF/MDAACsotahM2HChBx11FEZOnRokmTixIm56667cs011+S0005bZf1rrrkmCxYsyKOPPpoGDRokSbp06fLZpgYAAFiDWl26tnz58kyfPj0DBgz43x3Uq5cBAwbkscceq3GbO++8M/369cvw4cPTvn377LTTTjn33HOzcuXK1R5n2bJlWbRoUbUHAADA2qpV6Lz33ntZuXJl2rdvX215+/btM2/evBq3mTlzZm655ZasXLkyd999d84444xceOGF+fnPf77a44wbNy6tWrWqenTq1Kk2YwIAAJ9zdX7XtYqKirRr1y5XXHFFevfunYMPPjg/+9nPMnHixNVuM3r06CxcuLDqMWfOnLoeEwAAKJBafUanTZs2KS8vz/z586stnz9/fjp06FDjNh07dkyDBg1SXl5etaxHjx6ZN29eli9fnoYNG66yTaNGjdKoUaPajAYAAFClVmd0GjZsmN69e2fy5MlVyyoqKjJ58uT069evxm322GOPvP7666moqKha9uqrr6Zjx441Rg4AAMBnVetL10aOHJkrr7wy119/fV566aUce+yxWbx4cdVd2AYPHpzRo0dXrX/sscdmwYIFOeGEE/Lqq6/mrrvuyrnnnpvhw4evu1cBAADwb2p9e+mDDz447777bsaMGZN58+Zll112yT333FN1g4LZs2enXr3/7adOnTrl3nvvzUknnZRevXpliy22yAknnJBTTz113b0KAACAf1Pr0EmSESNGZMSIETU+N2XKlFWW9evXL48//vinORQAAECt1fld1wAAANY3oQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhfKrQueyyy9KlS5c0btw4u+++e6ZNm7ZW2910000pKyvLgQce+GkOCwAAsFZqHTqTJk3KyJEjM3bs2Dz99NPZeeedM3DgwLzzzjtr3G7WrFn5yU9+ki996UufelgAAIC1UevQmTBhQo466qgMHTo0O+ywQyZOnJimTZvmmmuuWe02K1euzGGHHZazzjor3bp1+0wDAwAAfJJahc7y5cszffr0DBgw4H93UK9eBgwYkMcee2y125199tlp165dhg0btlbHWbZsWRYtWlTtAQAAsLZqFTrvvfdeVq5cmfbt21db3r59+8ybN6/GbR555JFcffXVufLKK9f6OOPGjUurVq2qHp06darNmAAAwOdcnd517cMPP8wRRxyRK6+8Mm3atFnr7UaPHp2FCxdWPebMmVOHUwIAAEVTvzYrt2nTJuXl5Zk/f3615fPnz0+HDh1WWf+NN97IrFmzMmjQoKplFRUV/zxw/fp55ZVXsvXWW6+yXaNGjdKoUaPajAYAAFClVmd0GjZsmN69e2fy5MlVyyoqKjJ58uT069dvlfW33377PPfcc3n22WerHt/85jfz1a9+Nc8++6xL0gAAgDpRqzM6STJy5MgMGTIkffr0Sd++fXPxxRdn8eLFGTp0aJJk8ODB2WKLLTJu3Lg0btw4O+20U7XtW7dunSSrLAcAAFhXah06Bx98cN59992MGTMm8+bNyy677JJ77rmn6gYFs2fPTr16dfrRHwAAgDWqdegkyYgRIzJixIgan5syZcoat73uuus+zSEBAADWmlMvAABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACudThc5ll12WLl26pHHjxtl9990zbdq01a575ZVX5ktf+lI22WSTbLLJJhkwYMAa1wcAAPisah06kyZNysiRIzN27Ng8/fTT2XnnnTNw4MC88847Na4/ZcqUHHLIIXnooYfy2GOPpVOnTtlnn33yt7/97TMPDwAAUJNah86ECRNy1FFHZejQodlhhx0yceLENG3aNNdcc02N6//2t7/Ncccdl1122SXbb799rrrqqlRUVGTy5MmrPcayZcuyaNGiag8AAIC1VavQWb58eaZPn54BAwb87w7q1cuAAQPy2GOPrdU+lixZkhUrVmTTTTdd7Trjxo1Lq1atqh6dOnWqzZgAAMDnXK1C57333svKlSvTvn37asvbt2+fefPmrdU+Tj311Gy++ebVYuk/jR49OgsXLqx6zJkzpzZjAgAAn3P11+fBxo8fn5tuuilTpkxJ48aNV7teo0aN0qhRo/U4GQAAUCS1Cp02bdqkvLw88+fPr7Z8/vz56dChwxq3veCCCzJ+/Pg88MAD6dWrV+0nBQAAWEu1unStYcOG6d27d7UbCfzrxgL9+vVb7Xbnn39+zjnnnNxzzz3p06fPp58WAABgLdT60rWRI0dmyJAh6dOnT/r27ZuLL744ixcvztChQ5MkgwcPzhZbbJFx48YlSc4777yMGTMmN954Y7p06VL1WZ7mzZunefPm6/ClAAAA/FOtQ+fggw/Ou+++mzFjxmTevHnZZZddcs8991TdoGD27NmpV+9/TxRdfvnlWb58eb7zne9U28/YsWNz5plnfrbpAQAAavCpbkYwYsSIjBgxosbnpkyZUu3nWbNmfZpDAAAAfGq1/sJQAACADZ3QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHCEDgAAUDhCBwAAKByhAwAAFI7QAQAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAoHKEDAAAUjtABAAAKR+gAAACFI3QAAIDCEToAAEDhCB0AAKBwhA4AAFA4QgcAACgcoQMAABSO0AEAAApH6AAAAIUjdAAAgMIROgAAQOEIHQAAoHA+Vehcdtll6dKlSxo3bpzdd98906ZNW+P6N998c7bffvs0btw4PXv2zN133/2phgUAAFgbtQ6dSZMmZeTIkRk7dmyefvrp7Lzzzhk4cGDeeeedGtd/9NFHc8ghh2TYsGF55plncuCBB+bAAw/M888//5mHBwAAqEn92m4wYcKEHHXUURk6dGiSZOLEibnrrrtyzTXX5LTTTltl/V/96lfZd999c/LJJydJzjnnnNx///259NJLM3HixBqPsWzZsixbtqzq54ULFyZJFi1aVNtxNxgVy5bU6f4XlVXW6f6TZOU/Vtbp/j9aWbf735h/f4piY38f1PV7IPE++DzwPlizun4PJN4HpVbX74HE++CTbOzvgX/NX1n5Cf87V9bCsmXLKsvLyytvv/32assHDx5c+c1vfrPGbTp16lR50UUXVVs2ZsyYyl69eq32OGPHjq1M4uHh4eHh4eHh4eHhUeNjzpw5a2yXWp3Ree+997Jy5cq0b9++2vL27dvn5ZdfrnGbefPm1bj+vHnzVnuc0aNHZ+TIkVU/V1RUZMGCBdlss81SVlZWm5FZRxYtWpROnTplzpw5admyZanHgfXOewC8DyDxPtgQVFZW5sMPP8zmm2++xvVqfena+tCoUaM0atSo2rLWrVuXZhiqadmypTc1n2veA+B9AIn3Qam1atXqE9ep1c0I2rRpk/Ly8syfP7/a8vnz56dDhw41btOhQ4darQ8AAPBZ1Sp0GjZsmN69e2fy5MlVyyoqKjJ58uT069evxm369etXbf0kuf/++1e7PgAAwGdV60vXRo4cmSFDhqRPnz7p27dvLr744ixevLjqLmyDBw/OFltskXHjxiVJTjjhhOy111658MILs//+++emm27KU089lSuuuGLdvhLqVKNGjTJ27NhVLimEzwvvAfA+gMT7YGNSVln5SfdlW9Wll16aX/7yl5k3b1522WWXXHLJJdl9992TJF/5ylfSpUuXXHfddVXr33zzzTn99NMza9asdO/ePeeff37222+/dfYiAAAA/t2nCh0AAIANWa0+owMAALAxEDoAAEDhCB0AAKBwhA4AAFA4QgcAACgcoUONHnrooVKPAAAAn5rbS1OjRo0aZcstt8zQoUMzZMiQdOrUqdQjQcm89tpreeihh/LOO++koqKi2nNjxowp0VQAlNKcOXMyduzYXHPNNaUehdUQOtTovffeyw033JDrr78+L7zwQvr3759hw4blwAMPTMOGDUs9Hqw3V155ZY499ti0adMmHTp0SFlZWdVzZWVlefrpp0s4HdSNgw46aK3Xve222+pwEthwzZgxI7vttltWrlxZ6lFYDaHDJ3r66adz7bXX5ne/+12S5NBDD82wYcOy8847l3gyqHtbbbVVjjvuuJx66qmlHgXWm6FDh671utdee20dTgKlc+edd67x+ZkzZ2bUqFFCZwMmdFgrb7/9dq644oqMHz8+9evXz9KlS9OvX79MnDgxO+64Y6nHgzrTsmXLPPvss+nWrVupRwFgPapXr17Kysqypr8ql5WVCZ0NmJsRsForVqzILbfckv322y9bbbVV7r333lx66aWZP39+Xn/99Wy11Vb57ne/W+oxoU5997vfzX333VfqMQBYzzp27JjbbrstFRUVNT5curzhq1/qAdgw/fjHP87vfve7VFZW5ogjjsj555+fnXbaqer5Zs2a5YILLsjmm29ewimh7m2zzTY544wz8vjjj6dnz55p0KBBteePP/74Ek0GdWfXXXet9nm0NfGXPYqqd+/emT59eg444IAan/+ksz2UnkvXqNHee++dH/7whznooIPSqFGjGtf5+OOPM3Xq1Oy1117reTpYf7p27bra58rKyjJz5sz1OA2sH2edddZarzt27Ng6nARK4y9/+UsWLlyYxYsXZ999961xncWLF+epp57y96ANmNABAIB/U15enrlz56Zdu3bp1q1bnnzyyWy22WalHota8hkdajRu3Lga7wt/zTXX5LzzzivBRFB6lZWVLlMA+Bxo3bp13nzzzSTJrFmzVvkONTYOQoca/Z//83+y/fbbr7J8xx13zMSJE0swEZTO//zP/6Rnz55p0qRJmjRpkl69euWGG24o9ViwXqxcuTIXXHBB+vbtmw4dOmTTTTet9oAi+va3v5299torXbt2TVlZWfr06ZNu3brV+GDD5WYE1GjevHnp2LHjKsvbtm2buXPnlmAiKI0JEybkjDPOyIgRI7LHHnskSR555JEcc8wxee+993LSSSeVeEKoW2eddVauuuqqjBo1Kqeffnp+9rOfZdasWbnjjjsyZsyYUo8HdeKKK67IQQcdlNdffz3HH398jjrqqLRo0aLUY1FLPqNDjbp3756xY8fm8MMPr7b8hhtuyNixY30Am8+Nrl275qyzzsrgwYOrLb/++utz5plnVl3aAEW19dZb55JLLsn++++fFi1a5Nlnn61a9vjjj+fGG28s9YhQp4YOHZpLLrlE6GyEnNGhRkcddVROPPHErFixIv3790+STJ48OaecckpGjRpV4ulg/Zk7d26++MUvrrL8i1/8orObfC7MmzcvPXv2TJI0b948CxcuTJJ84xvfyBlnnFHK0WC9uPbaa0s9Ap+S0KFGJ598ct5///0cd9xxWb58eZKkcePGOfXUUzN69OgSTwfrzzbbbJPf//73+elPf1pt+aRJk9K9e/cSTQXrz5Zbbpm5c+emc+fO2XrrrXPfffdlt912y5NPPrnarx8A2BC4dI01+uijj/LSSy+lSZMm6d69u3+p8blz66235uCDD86AAQOqPqMzderUTJ48Ob///e/zrW99q8QTQt067bTT0rJly/z0pz/NpEmTcvjhh6dLly6ZPXt2TjrppIwfP77UIwLUSOgAfILp06fnoosuyksvvZQk6dGjR0aNGpVdd921xJPB+vf444/n0UcfTffu3TNo0KBSjwOwWkKH1Xrqqafy+9//PrNnz666fO1fbrvtthJNBQAAn8z36FCjm266KV/84hfz0ksv5fbbb8+KFSvywgsv5MEHH0yrVq1KPR7UqUWLFlX785oeUHS+QBrYWDmjQ4169eqVH/3oRxk+fHhatGiRGTNmpGvXrvnRj36Ujh075qyzzir1iFBnysvLM3fu3LRr1y716tVLWVnZKutUVlamrKwsK1euLMGEsP506dIlN9544yp3H3ziiSfy/e9/3y3WgQ2Wu65RozfeeCP7779/kqRhw4ZZvHhxysrKctJJJ6V///5Ch0J78MEHq77x/aGHHirxNFBavkAa2FgJHWq0ySab5MMPP0ySbLHFFnn++efTs2fPfPDBB1myZEmJp4O6tddee9X4Z/g86tSpU6ZOnZquXbtWWz516tRsvvnmJZoK4JP5jA41+vKXv5z7778/SfLd7343J5xwQo466qgccsgh2XvvvUs8Haw/99xzTx555JGqny+77LLssssuOfTQQ/P3v/+9hJPB+vGvL5C+9tpr89Zbb+Wtt97KNddck5NOOilHHXVUqccDWC2f0aFGCxYsyNKlS7P55punoqIi559/ftXtRE8//fRssskmpR4R1ouePXvmvPPOy3777Zfnnnsuffr0yahRo/LQQw9l++23943ZFF5lZWVOO+20XHLJJat8gfSYMWNKPB3A6gkdVvHxxx/nxhtvzMCBA9O+fftSjwMl1bx58zz//PPp0qVLzjzzzDz//PO55ZZb8vTTT2e//fbLvHnzSj0irBe+QBrY2Lh0jVXUr18/xxxzTJYuXVrqUaDkGjZsWPW5tAceeCD77LNPkmTTTTd1e2k+V+bNm5cFCxZk6623TqNGjeK/kwIbOqFDjfr27Ztnn3221GNAye25554ZOXJkzjnnnEybNq3qboSvvvpqttxyyxJPB3Xv/fffz957751tt902++23X9Wd1oYNG5ZRo0aVeDqA1RM61Oi4447LyJEjc+mll+axxx7LX/7yl2oP+Ly49NJLU79+/dxyyy25/PLLs8UWWyRJ/vjHP2bfffct8XRQ90466aQ0aNAgs2fPTtOmTauWH3zwwbnnnntKOBnAmvmMDjWqV2/VBi4rK/MliQCfMx06dMi9996bnXfeueoLpLt165aZM2emV69e+eijj0o9IkCNfI8ONfJN13yeLVq0KC1btqz685r8az0oqsWLF1c7k/MvCxYscEMCYIPmjA7AfygvL8/cuXPTrl271KtXL2VlZaus4+wmnxf77bdfevfunXPOOSctWrTIX/7yl2y11Vb5/ve/n4qKitxyyy2lHhGgRs7oUKP/+Z//WePzgwcPXk+TwPr34IMPZtNNN02SPPTQQyWeBkrrl7/8Zfr375+nnnoqy5cvzymnnJIXXnghCxYsyNSpU0s9HsBqOaNDjf7zC0FXrFiRJUuWpGHDhmnatGkWLFhQoskAWF9WrFiRfffdN+PGjcv999+fGTNm5KOPPspuu+2W4cOHp2PHjqUeEWC1hA5r7bXXXsuxxx6bk08+OQMHDiz1OLBeXHvttWnevHm++93vVlt+8803Z8mSJRkyZEiJJoP1o23btnn00UfTvXv3Uo8CUCtuL81a6969e8aPH58TTjih1KPAejNu3Li0adNmleXt2rXLueeeW4KJYP06/PDDc/XVV5d6DIBa8xkdaqV+/fp5++23Sz0GrDezZ89O165dV1m+1VZbZfbs2SWYCNavjz/+ONdcc00eeOCB9O7dO82aNav2/IQJE0o0GcCaCR1qdOedd1b7ubKyMnPnzs2ll16aPfbYo0RTwfrXrl27/OUvf0mXLl2qLZ8xY0Y222yz0gwF69Hzzz+f3XbbLUny6quvVnuupjsSAmwohA41OvDAA6v9XFZWlrZt26Z///658MILSzMUlMAhhxyS448/Pi1atMiXv/zlJMmf/vSnnHDCCfn+979f4umg7rnzILCxcjMCgDVYvnx5jjjiiNx8882pX/+f/22ooqIigwcPzsSJE9OwYcMSTwgA1EToAKyFV199NTNmzEiTJk3Ss2fPbLXVVqUeCQBYA5euUaNvf/vb6du3b0499dRqy88///w8+eSTufnmm0s0GZRGly5dUllZma233rrqzA4AsOFye2lq9PDDD2e//fZbZfnXv/71PPzwwyWYCEpjyZIlGTZsWJo2bZodd9yx6k5rP/7xjzN+/PgSTwcArI7QoUYfffRRjZ89aNCgQRYtWlSCiaA0Ro8enRkzZmTKlClp3Lhx1fIBAwZk0qRJJZwMAFgToUONevbsWeNf4m666abssMMOJZgISuOOO+7IpZdemj333LParXR33HHHvPHGGyWcDABYExeaU6MzzjgjBx10UN544430798/STJ58uT87ne/8/kcPlfefffdtGvXbpXlixcv9h0iALABc0aHGg0aNCh33HFHXn/99Rx33HEZNWpU/vrXv+aBBx5Y5Tt2oMj69OmTu+66q+rnf8XNVVddlX79+pVqLADgE7i9NMAaPPLII/n617+eww8/PNddd11+9KMf5cUXX8yjjz6aP/3pT+ndu3epRwQAauCMDjV68skn88QTT6yy/IknnshTTz1VgomgNPbcc8/MmDEjH3/8cXr27Jn77rsv7dq1y2OPPSZyAGADJnSo0fDhwzNnzpxVlv/tb3/L8OHDSzARrH8rVqzIkUcembKyslx55ZWZNm1aXnzxxfzmN79Jz549Sz0eALAGQocavfjii9ltt91WWb7rrrvmxRdfLMFEsP41aNAgt956a6nHAAA+BaFDjRo1apT58+evsnzu3Lm+FZ7PlQMPPDB33HFHqccAAGrJ31ip0T777JPRo0fnD3/4Q1q1apUk+eCDD/LTn/40X/va10o8Haw/3bt3z9lnn52pU6emd+/eadasWbXnjz/++BJNBgCsibuuUaO//e1v+fKXv5z3338/u+66a5Lk2WefTfv27XP//fenU6dOJZ4Q1o+uXbuu9rmysrLMnDlzPU4DAKwtocNqLV68OL/97W8zY8aMNGnSJL169cohhxySBg0alHo0KIl//d+lLwoFgA2f0GGNXnzxxcyePTvLly+vtvyb3/xmiSaC9e/qq6/ORRddlNdeey3JPy9nO/HEE/PDH/6wxJMBAKvjMzrUaObMmfnWt76V5557LmVlZamsrKz2X7FXrlxZwulg/RkzZkwmTJiQH//4x+nXr1+S5LHHHstJJ52U2bNn5+yzzy7xhABATZzRoUaDBg1KeXl5rrrqqnTt2jVPPPFEFixYkFGjRuWCCy7Il770pVKPCOtF27Ztc8kll+SQQw6ptvx3v/tdfvzjH+e9994r0WQAwJo4o0ONHnvssTz44INp06ZN6tWrl/Ly8uy5554ZN25cjj/++DzzzDOlHhHWixUrVqRPnz6rLO/du3c+/vjjEkwEAKwN36NDjVauXJkWLVokSdq0aZO33347SbLVVlvllVdeKeVosF4dccQRufzyy1dZfsUVV+Swww4rwUQAwNpwRoca7bTTTpkxY0a6du2a3XffPeeff34aNmyYK664It26dSv1eLBeXX311bnvvvvyhS98IUnyxBNPZPbs2Rk8eHBGjhxZtd6ECRNKNSIA8B98Roca3XvvvVm8eHEOOuigvP766/nGN76RV199NZtttlkmTZqU/v37l3pEWC+++tWvrtV6ZWVlefDBB+t4GgBgbQkd1tqCBQuyySab+A4RAAA2eEIHAAAoHDcjAAAACkfoAAAAhSN0AACAwhE6AABA4QgdAACgcIQOAABQOEIHAAAonP8P9mBw/Ct7rMQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "compare_metrics = pd.DataFrame({\"baseline\": baseline_metrics,\n",
    "                                \"clf_2\": clf_2_metrics,\n",
    "                                \"random search\": rs_metrics,\n",
    "                                \"grid search\": gs_metrics})\n",
    "compare_metrics.plot.bar(figsize=(10, 8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "After trying many different combinations of hyperparamters, we get a slight improvement in results.\n",
    "\n",
    "However, sometimes you'll notice that your results don't change much.\n",
    "\n",
    "These things might happen. \n",
    "\n",
    "But it's important to remember, it's not over. There more things you can try.\n",
    "\n",
    "In a hyperparameter tuning sense, there may be a better set we could find through more extensive searching with `RandomizedSearchCV` and `GridSearchCV`, this would require more experimentation.\n",
    "\n",
    "Other techniques you could:\n",
    "* **Collecting more data** - Based on the results our models are getting now, it seems like they're very capable of finding patterns. Collecting more data may improve a models ability to find patterns. However, your ability to do this will largely depend on the project you're working on.\n",
    "* **Try a more advanced model** - Although our tuned Random Forest model is doing pretty well, a more advanced ensemble method such as [XGBoost](https://xgboost.ai/) or [CatBoost](https://catboost.ai/) might perform better. I'll leave these for extra-curriculum.\n",
    "\n",
    "Since machine learning is part engineering, part science, these kind of experiments are common place in any machine learning project.\n",
    "\n",
    "Now we've got a tuned Random Forest model, let's find out how we might save it and export it so we can share it with others or potentially use it in an external application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Saving and loading trained machine learning models\n",
    "\n",
    "Our `GridSearchCV` model (`gs_clf`) has the best results so far, we'll export it and save it to file.\n",
    "\n",
    "### 6.1 Saving and loading a model with `pickle`\n",
    "\n",
    "We saw right at the start, one way to save a model is using Python's [`pickle` module](https://docs.python.org/3/library/pickle.html).\n",
    "\n",
    "We'll use `pickle`'s `dump()` method and pass it our model, `gs_clf`, along with the `open()` function containing a string for the filename we want to save our model as, along with the `\"wb\"` string which stands for \"write binary\", which is the file type `open()` will write our model as."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Save an existing model to file\n",
    "best_model_file_name_pickle = \"gs_random_forest_model_1.pkl\" # .pkl extension stands for \"pickle\"\n",
    "pickle.dump(gs_clf, open(best_model_file_name_pickle, \"wb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's saved, we can import it using `pickle`'s `load()` function, passing it `open()` containing the filename as a string and `\"rb\"` standing for \"read binary\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a saved model\n",
    "loaded_pickle_model = pickle.load(open(best_model_file_name_pickle, \"rb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've reimported your trained model using `pickle`, you can use it to make predictions as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 88.52%\n",
      "Precision: 0.88\n",
      "Recall: 0.91\n",
      "F1 score: 0.89\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.89,\n",
       " 'precision': np.float64(0.88),\n",
       " 'recall': np.float64(0.91),\n",
       " 'f1': np.float64(0.89)}"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make predictions and evaluate the loaded model\n",
    "pickle_y_preds = loaded_pickle_model.predict(X_test)\n",
    "loaded_pickle_model_metrics = evaluate_preds(y_test, pickle_y_preds)\n",
    "loaded_pickle_model_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice the reimported model evaluation metrics are the same as the model before we exported it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_pickle_model_metrics == gs_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Saving and loading a model with [`joblib`](https://joblib.readthedocs.io/en/latest/persistence.html)\n",
    "\n",
    "The other way to load and save models is with `joblib`. Which works relatively the same as `pickle`.\n",
    "\n",
    "To save a model, we can use `joblib`'s `dump()` function, passing it the model (`gs_clf`) and the desired filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['gs_random_forest_model_1.joblib']"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from joblib import dump, load\n",
    "\n",
    "# Save a model to file\n",
    "best_model_file_name_joblib = \"gs_random_forest_model_1.joblib\"\n",
    "dump(gs_clf, filename=best_model_file_name_joblib) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've saved a model using `dump()`, you can import it using `load()` and passing it the filename of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a saved joblib model\n",
    "loaded_joblib_model = load(filename=best_model_file_name_joblib)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, once imported, we can make predictions with our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acc: 88.52%\n",
      "Precision: 0.88\n",
      "Recall: 0.91\n",
      "F1 score: 0.89\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.89,\n",
       " 'precision': np.float64(0.88),\n",
       " 'recall': np.float64(0.91),\n",
       " 'f1': np.float64(0.89)}"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make and evaluate joblib predictions \n",
    "joblib_y_preds = loaded_joblib_model.predict(X_test)\n",
    "loaded_joblib_model_metrics = evaluate_preds(y_test, joblib_y_preds)\n",
    "loaded_joblib_model_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And once again, you'll notice the evaluation metrics are the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_joblib_model_metrics == gs_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So which one should you use, `pickle` or `joblib`?\n",
    "\n",
    "According to [Scikit-Learn's model persistence documentation](https://scikit-learn.org/stable/model_persistence.html), they suggest it may be more efficient to use `joblib` as it's more efficient with large numpy arrays (which is what may be contained in trained/fitted Scikit-Learn models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Revisiting the entire pipeline\n",
    "\n",
    "We've covered a lot. And so far, it seems to be all over the place, which it is.\n",
    "\n",
    "But not to worry, machine learning projects often start out like this. \n",
    "\n",
    "A whole bunch of experimenting and code all over the place at the start and then once you've found something which works, the refinement process begins.\n",
    "\n",
    "What would this refinement process look like?\n",
    "\n",
    "We'll use the car sales regression problem (predicting the sale price of cars) as an example.\n",
    "\n",
    "To tidy things up, we'll be using Scikit-Learn's [`sklearn.pipeline.Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) class. \n",
    "\n",
    "You can imagine `Pipeline` as being a way to string a number of different Scikit-Learn processes together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 Creating a regression [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html)\n",
    "You might recall when, way back in Section 2: Getting Data Ready, we dealt with the car sales data, to build a regression model on it, we had to encode the categorical features into numbers and fill the missing data.\n",
    "\n",
    "The code we used worked, but it was a bit all over the place. \n",
    "\n",
    "Good news is, `Pipeline` can help us clean it up.\n",
    "\n",
    "Let's remind ourselves what our [`car-sales-extended-missing-data.csv`](https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/car-sales-extended-missing-data.csv) looks like in DataFrame form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Colour</th>\n",
       "      <th>Odometer (KM)</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>35431.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15323.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>Blue</td>\n",
       "      <td>192714.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>19943.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Honda</td>\n",
       "      <td>White</td>\n",
       "      <td>84714.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28343.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Toyota</td>\n",
       "      <td>White</td>\n",
       "      <td>154365.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>13434.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nissan</td>\n",
       "      <td>Blue</td>\n",
       "      <td>181577.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>14043.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Make Colour  Odometer (KM)  Doors    Price\n",
       "0   Honda  White        35431.0    4.0  15323.0\n",
       "1     BMW   Blue       192714.0    5.0  19943.0\n",
       "2   Honda  White        84714.0    4.0  28343.0\n",
       "3  Toyota  White       154365.0    4.0  13434.0\n",
       "4  Nissan   Blue       181577.0    3.0  14043.0"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data = pd.read_csv(\"../data/car-sales-extended-missing-data.csv\") # load from local directory\n",
    "data = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv\") # load directly from URL\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              object\n",
       "Colour            object\n",
       "Odometer (KM)    float64\n",
       "Doors            float64\n",
       "Price            float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             49\n",
       "Colour           50\n",
       "Odometer (KM)    50\n",
       "Doors            50\n",
       "Price            50\n",
       "dtype: int64"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's 1000 rows, three features are categorical (`Make`, `Colour`, `Doors`), the other two are numerical (`Odometer (KM)`, `Price`) and there's 249 missing values.\n",
    "\n",
    "We're going to have to turn the categorical features into numbers and fill the missing values before we can fit a model.\n",
    "\n",
    "We'll build a `Pipeline` to do so.\n",
    "\n",
    "`Pipeline`'s main input parameter is `steps` which is a list of tuples (`[(step_name, action_to_take)]`) of the step name, plus the action you'd like it to perform.\n",
    "\n",
    "In our case, you could think of the steps as:\n",
    "1. Fill missing data\n",
    "2. Convert data to numbers\n",
    "3. Build a model on the data\n",
    "\n",
    "Let's do it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22188417408787875"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting data ready\n",
    "import pandas as pd\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Modelling\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "\n",
    "# Setup random seed\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "\n",
    "# Import data and drop the rows with missing labels\n",
    "data = pd.read_csv(\"https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/car-sales-extended-missing-data.csv\")\n",
    "data.dropna(subset=[\"Price\"], inplace=True)\n",
    "\n",
    "# Define different features and transformer pipelines\n",
    "categorical_features = [\"Make\", \"Colour\"]\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"missing\")),\n",
    "    (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\"))])\n",
    "\n",
    "door_feature = [\"Doors\"]\n",
    "door_transformer = Pipeline(steps=[\n",
    "    (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=4))])\n",
    "\n",
    "numeric_features = [\"Odometer (KM)\"]\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "    (\"imputer\", SimpleImputer(strategy=\"mean\"))\n",
    "])\n",
    "\n",
    "# Setup preprocessing steps (fill missing values, then convert to numbers)\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        (\"cat\", categorical_transformer, categorical_features),\n",
    "        (\"door\", door_transformer, door_feature),\n",
    "        (\"num\", numeric_transformer, numeric_features)])\n",
    "\n",
    "# Create a preprocessing and modelling pipeline\n",
    "model = Pipeline(steps=[(\"preprocessor\", preprocessor),\n",
    "                        (\"model\", RandomForestRegressor(n_jobs=-1))])\n",
    "\n",
    "# Split data\n",
    "X = data.drop(\"Price\", axis=1)\n",
    "y = data[\"Price\"]\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "# Fit and score the model\n",
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we've done is combine a series of data preprocessing steps (filling missing values, encoding numerical values) as well as a model into a `Pipeline`.\n",
    "\n",
    "Doing so not only cleans up the code, it ensures the same steps are taken every time the code is run rather than having multiple different processing steps happening in different stages.\n",
    "\n",
    "It's also possible to `GridSearchCV` or `RandomizedSearchCV` with a `Pipeline`.\n",
    "\n",
    "The main difference is when creating a hyperparameter grid, you have to add a prefix to each hyperparameter (see the [documentation for `RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) for a full list of possible hyperparameters to tune).\n",
    "\n",
    "The prefix is the name of the `Pipeline` step you'd like to alter, followed by two underscores.\n",
    "\n",
    "For example, to adjust `n_estimators` of `\"model\"` in the `Pipeline`, you'd use: `\"model__n_estimators\"` (note the double underscore after `model__` at the start).\n",
    "\n",
    "Let's see it!\n",
    "\n",
    "> **Note:** Depending on your computer's processing power, the cell below may take a few minutes to run. For reference, it took about ~60 seconds on my M1 Pro MacBook Pro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 16 candidates, totalling 80 fits\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=None, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.7s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=2, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=mean; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=100, preprocessor__num__imputer__strategy=median; total time=   0.1s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=mean; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.6s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n",
      "[CV] END model__max_depth=5, model__max_features=sqrt, model__min_samples_split=4, model__n_estimators=1000, preprocessor__num__imputer__strategy=median; total time=   0.5s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-4 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-4 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-4 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-4 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-4 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-4 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-4 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-4 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-4 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-4 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-4 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-4 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-4 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-4 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                                        ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                                                        strategy=&#x27;constant&#x27;)),\n",
       "                                                                                         (&#x27;onehot&#x27;,\n",
       "                                                                                          OneHotEncoder(handle_unknown=&#x27;ignore&#x27;))]),\n",
       "                                                                         [&#x27;Make&#x27;,\n",
       "                                                                          &#x27;Colour&#x27;]),\n",
       "                                                                        (&#x27;door&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer(fill_value=4,\n",
       "                                                                                                        strategy=&#x27;constant&#x27;))]),\n",
       "                                                                         [&#x27;Doors&#x27;]),\n",
       "                                                                        (&#x27;num&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer())]),\n",
       "                                                                         [&#x27;Odometer &#x27;\n",
       "                                                                          &#x27;(KM)&#x27;])])),\n",
       "                                       (&#x27;model&#x27;,\n",
       "                                        RandomForestRegressor(n_jobs=-1))]),\n",
       "             param_grid={&#x27;model__max_depth&#x27;: [None, 5],\n",
       "                         &#x27;model__max_features&#x27;: [&#x27;sqrt&#x27;],\n",
       "                         &#x27;model__min_samples_split&#x27;: [2, 4],\n",
       "                         &#x27;model__n_estimators&#x27;: [100, 1000],\n",
       "                         &#x27;preprocessor__num__imputer__strategy&#x27;: [&#x27;mean&#x27;,\n",
       "                                                                  &#x27;median&#x27;]},\n",
       "             verbose=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;GridSearchCV<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.model_selection.GridSearchCV.html\">?<span>Documentation for GridSearchCV</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                                        ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                                                        strategy=&#x27;constant&#x27;)),\n",
       "                                                                                         (&#x27;onehot&#x27;,\n",
       "                                                                                          OneHotEncoder(handle_unknown=&#x27;ignore&#x27;))]),\n",
       "                                                                         [&#x27;Make&#x27;,\n",
       "                                                                          &#x27;Colour&#x27;]),\n",
       "                                                                        (&#x27;door&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer(fill_value=4,\n",
       "                                                                                                        strategy=&#x27;constant&#x27;))]),\n",
       "                                                                         [&#x27;Doors&#x27;]),\n",
       "                                                                        (&#x27;num&#x27;,\n",
       "                                                                         Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                                          SimpleImputer())]),\n",
       "                                                                         [&#x27;Odometer &#x27;\n",
       "                                                                          &#x27;(KM)&#x27;])])),\n",
       "                                       (&#x27;model&#x27;,\n",
       "                                        RandomForestRegressor(n_jobs=-1))]),\n",
       "             param_grid={&#x27;model__max_depth&#x27;: [None, 5],\n",
       "                         &#x27;model__max_features&#x27;: [&#x27;sqrt&#x27;],\n",
       "                         &#x27;model__min_samples_split&#x27;: [2, 4],\n",
       "                         &#x27;model__n_estimators&#x27;: [100, 1000],\n",
       "                         &#x27;preprocessor__num__imputer__strategy&#x27;: [&#x27;mean&#x27;,\n",
       "                                                                  &#x27;median&#x27;]},\n",
       "             verbose=2)</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">best_estimator_: Pipeline</label><div class=\"sk-toggleable__content fitted\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                   SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                                 strategy=&#x27;constant&#x27;)),\n",
       "                                                                  (&#x27;onehot&#x27;,\n",
       "                                                                   OneHotEncoder(handle_unknown=&#x27;ignore&#x27;))]),\n",
       "                                                  [&#x27;Make&#x27;, &#x27;Colour&#x27;]),\n",
       "                                                 (&#x27;door&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                   SimpleImputer(fill_value=4,\n",
       "                                                                                 strategy=&#x27;constant&#x27;))]),\n",
       "                                                  [&#x27;Doors&#x27;]),\n",
       "                                                 (&#x27;num&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                   SimpleImputer())]),\n",
       "                                                  [&#x27;Odometer (KM)&#x27;])])),\n",
       "                (&#x27;model&#x27;,\n",
       "                 RandomForestRegressor(max_depth=5, max_features=&#x27;sqrt&#x27;,\n",
       "                                       n_jobs=-1))])</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;preprocessor: ColumnTransformer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.compose.ColumnTransformer.html\">?<span>Documentation for preprocessor: ColumnTransformer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehot&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;))]),\n",
       "                                 [&#x27;Make&#x27;, &#x27;Colour&#x27;]),\n",
       "                                (&#x27;door&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=4,\n",
       "                                                                strategy=&#x27;constant&#x27;))]),\n",
       "                                 [&#x27;Doors&#x27;]),\n",
       "                                (&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;, SimpleImputer())]),\n",
       "                                 [&#x27;Odometer (KM)&#x27;])])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">cat</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;Make&#x27;, &#x27;Colour&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;SimpleImputer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.impute.SimpleImputer.html\">?<span>Documentation for SimpleImputer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div> </div></div><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;OneHotEncoder<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.OneHotEncoder.html\">?<span>Documentation for OneHotEncoder</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div> </div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">door</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;Doors&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;SimpleImputer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.impute.SimpleImputer.html\">?<span>Documentation for SimpleImputer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>SimpleImputer(fill_value=4, strategy=&#x27;constant&#x27;)</pre></div> </div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">num</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;Odometer (KM)&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;SimpleImputer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.impute.SimpleImputer.html\">?<span>Documentation for SimpleImputer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>SimpleImputer()</pre></div> </div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RandomForestRegressor<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.ensemble.RandomForestRegressor.html\">?<span>Documentation for RandomForestRegressor</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RandomForestRegressor(max_depth=5, max_features=&#x27;sqrt&#x27;, n_jobs=-1)</pre></div> </div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('preprocessor',\n",
       "                                        ColumnTransformer(transformers=[('cat',\n",
       "                                                                         Pipeline(steps=[('imputer',\n",
       "                                                                                          SimpleImputer(fill_value='missing',\n",
       "                                                                                                        strategy='constant')),\n",
       "                                                                                         ('onehot',\n",
       "                                                                                          OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                                         ['Make',\n",
       "                                                                          'Colour']),\n",
       "                                                                        ('door',\n",
       "                                                                         Pipeline(steps=[('imputer',\n",
       "                                                                                          SimpleImputer(fill_value=4,\n",
       "                                                                                                        strategy='constant'))]),\n",
       "                                                                         ['Doors']),\n",
       "                                                                        ('num',\n",
       "                                                                         Pipeline(steps=[('imputer',\n",
       "                                                                                          SimpleImputer())]),\n",
       "                                                                         ['Odometer '\n",
       "                                                                          '(KM)'])])),\n",
       "                                       ('model',\n",
       "                                        RandomForestRegressor(n_jobs=-1))]),\n",
       "             param_grid={'model__max_depth': [None, 5],\n",
       "                         'model__max_features': ['sqrt'],\n",
       "                         'model__min_samples_split': [2, 4],\n",
       "                         'model__n_estimators': [100, 1000],\n",
       "                         'preprocessor__num__imputer__strategy': ['mean',\n",
       "                                                                  'median']},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Using grid search with pipeline\n",
    "pipe_grid = {\n",
    "    \"preprocessor__num__imputer__strategy\": [\"mean\", \"median\"], # note the double underscore after each prefix \"preprocessor__\"\n",
    "    \"model__n_estimators\": [100, 1000],\n",
    "    \"model__max_depth\": [None, 5],\n",
    "    \"model__max_features\": [\"sqrt\"],\n",
    "    \"model__min_samples_split\": [2, 4]\n",
    "}\n",
    "\n",
    "gs_model = GridSearchCV(model, pipe_grid, cv=5, verbose=2)\n",
    "gs_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's find the score of our model (by default `GridSearchCV` saves the best model to the `gs_model` object)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2848784564026805"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Score the best model\n",
    "gs_model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! \n",
    "\n",
    "Using `GridSearchCV` we see a nice boost in our models score. \n",
    "\n",
    "And the best thing is, because it's all in a `Pipeline`, we could easily replicate these results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where to next?\n",
    "\n",
    "If you've made it this far, congratulations! We've covered a lot of ground in the Scikit-Learn library.\n",
    "\n",
    "As you might've guessed, there's a lot more to be discovered.\n",
    "\n",
    "But for the time being, you should be equipped with some of the most useful features of the library to start trying to apply them to your own problems.\n",
    "\n",
    "Somewhere you might like to look next is to apply what you've learned above to a [Kaggle competition](https://www.kaggle.com). \n",
    "\n",
    "Kaggle competitions are great places to practice your data science and machine learning skills and compare your results with others.\n",
    "\n",
    "A great idea would be to try to combine the heart disease classification code, as well as the `Pipeline` code, to build a model for the [Titanic dataset](https://www.kaggle.com/c/titanic).\n",
    "\n",
    "Otherwise, if you'd like to figure out what else the Scikit-Learn library is capable of I'd highly recommend browsing through the [Scikit-Learn User Guide](https://scikit-learn.org/stable/user_guide.html) and seeing what sparks your interest.\n",
    "\n",
    "Finally, as an extra-curriculum extension, you might want to look into trying out the [CatBoost library](https://catboost.ai) for dealing with non-numerical data automatically. \n",
    "\n",
    "The CatBoost algorithm is advanced version of a decision tree (like a Random Forest with superpowers) and is [used in production at several large technology companies](https://blog.cloudflare.com/how-cloudflare-runs-ml-inference-in-microseconds/), including Cloudflare."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "c556f6eb84b27a92005489cdcf9c9b80cc62ee891441f20eabfc5ad7282165a3"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}