09 - Differentially Private Training with Opacus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with DP-SGD using Opacus\n",
    "\n",
    "In this notebook, we'll explore using DP-SGD using the Opacus library from Meta/PyTorch!\n",
    "\n",
    "The problem we will try to solve is determining the review score from looking at a Steam review. The dataset can be [downloaded from Kaggle here](https://www.kaggle.com/datasets/luthfim/steam-reviews-dataset?resource=download). This dataset has not gone through any privacy-preserving preprocessing (like many datasets we can find on Kaggle and in other machine learning communities)!\n",
    "\n",
    "We would like to use a BERT base and fine-tune it using Opacus to add differential privacy.\n",
    "\n",
    "This notebook is based on the [tutorial notebook from Opacus](https://opacus.ai/tutorials/building_text_classifier). They also have other tutorials available for tasks like image recognition and recommendors!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import torch\n",
    "import transformers\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "from torch.utils.data import DataLoader, SequentialSampler\n",
    "from transformers import BertConfig, BertTokenizer, BertForSequenceClassification\n",
    "from datasets import Dataset\n",
    "\n",
    "from opacus import PrivacyEngine\n",
    "from opacus.utils.uniform_sampler import UniformWithReplacementSampler\n",
    "from opacus.utils.batch_memory_manager import BatchMemoryManager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to get BERT and set what layers we actually want to train with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model_name = \"bert-base-cased\"\n",
    "config = BertConfig.from_pretrained(\n",
    "    model_name,\n",
    "    num_labels=3,\n",
    ")\n",
    "tokenizer = BertTokenizer.from_pretrained(\n",
    "    \"bert-base-cased\",\n",
    "    do_lower_case=False,\n",
    "    padding=True,\n",
    ")\n",
    "model = BertForSequenceClassification.from_pretrained(\n",
    "    \"bert-base-cased\",\n",
    "    config=config,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total parameters count: 108312579\n",
      "Trainable parameters count: 7680771\n"
     ]
    }
   ],
   "source": [
    "trainable_layers = [model.bert.encoder.layer[-1], model.bert.pooler, model.classifier]\n",
    "total_params = 0\n",
    "trainable_params = 0\n",
    "\n",
    "for p in model.parameters():\n",
    "        p.requires_grad = False\n",
    "        total_params += p.numel()\n",
    "\n",
    "for layer in trainable_layers:\n",
    "    for p in layer.parameters():\n",
    "        p.requires_grad = True\n",
    "        trainable_params += p.numel()\n",
    "\n",
    "print(f\"Total parameters count: {total_params}\") # ~108M\n",
    "print(f\"Trainable parameters count: {trainable_params}\") # ~7M\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's take a look at the distribution of labels in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date_posted</th>\n",
       "      <th>funny</th>\n",
       "      <th>helpful</th>\n",
       "      <th>hour_played</th>\n",
       "      <th>is_early_access_review</th>\n",
       "      <th>recommendation</th>\n",
       "      <th>review</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-10</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>578</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>&amp;gt Played as German Reich&amp;gt Declare war on B...</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>184</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>yes.</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-07</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>892</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Very good game although a bit overpriced in my...</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-06-14</td>\n",
       "      <td>126</td>\n",
       "      <td>1086</td>\n",
       "      <td>676</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Out of all the reviews I wrote This one is pro...</td>\n",
       "      <td>Dead by Daylight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-06-20</td>\n",
       "      <td>85</td>\n",
       "      <td>2139</td>\n",
       "      <td>612</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Disclaimer I survivor main. I play games for f...</td>\n",
       "      <td>Dead by Daylight</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  date_posted  funny  helpful  hour_played  is_early_access_review  \\\n",
       "0  2019-02-10      2        4          578                   False   \n",
       "1  2019-02-10      0        0          184                   False   \n",
       "2  2019-02-07      0        0          892                   False   \n",
       "3  2018-06-14    126     1086          676                   False   \n",
       "4  2017-06-20     85     2139          612                   False   \n",
       "\n",
       "  recommendation                                             review  \\\n",
       "0    Recommended  &gt Played as German Reich&gt Declare war on B...   \n",
       "1    Recommended                                               yes.   \n",
       "2    Recommended  Very good game although a bit overpriced in my...   \n",
       "3    Recommended  Out of all the reviews I wrote This one is pro...   \n",
       "4    Recommended  Disclaimer I survivor main. I play games for f...   \n",
       "\n",
       "                                         title  \n",
       "0  Expansion - Hearts of Iron IV: Man the Guns  \n",
       "1  Expansion - Hearts of Iron IV: Man the Guns  \n",
       "2  Expansion - Hearts of Iron IV: Man the Guns  \n",
       "3                             Dead by Daylight  \n",
       "4                             Dead by Daylight  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/steam_reviews.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Recommended        303593\n",
       "Not Recommended    131298\n",
       "Name: recommendation, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.recommendation.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['labels'] = df.recommendation.map(lambda x: 1 if x=='Recommended' else 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a fairly significant skew in this set to recommend, let's hope we can still find some useful information! \n",
    "\n",
    "We also need to tell PyTorch the max string length for the review to appropriately use the transformer. Once we have that, we can start training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.review = df.review.fillna('')\n",
    "df['review_len'] = df.review.map(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: >"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjoAAAGdCAYAAAAbudkLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABH3klEQVR4nO3df1xT970/8FdEiEghhVKIqdS6W+t0WHcv7RTbTq2Kev3Rrr3XtvTytXfOzlq1TF1X2211vZs4tbpurtb+mPaHla7zV1stBauiCIiiKIhSbVF+SAAlJICQ8OPz/YNyJCGEJCQkOb6ej0ceD8h5J/kk5+ScVz7nc85RCCEEiIiIiGSon6cbQEREROQuDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkW/093QBPamtrw5UrVxAcHAyFQuHp5hAREZEdhBCoq6uDRqNBv362+2xu6qBz5coVREVFeboZRERE5ITS0lIMHjzYZs1NHXSCg4MBtH9QISEhHm4NERER2cNgMCAqKkrajttyUwedjt1VISEhDDpEREQ+xp5hJxyMTERERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0REVpXWXMfm9G9R19Ts6aYQOe2mvno5ERF1b/obR1BvbMHFqnqs/e/Rnm4OkVPYo0NERFbVG1sAANnF1zzcEiLnMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBGRTQooPN0EIqcx6BARkU0K5hzyYQ4FnU2bNuHee+9FSEgIQkJCEBsbiy+//FKaLoTAypUrodFoEBgYiAkTJuDs2bNmz2E0GrF48WKEh4cjKCgIs2fPRllZmVmNTqdDQkICVCoVVCoVEhISUFtba1ZTUlKCWbNmISgoCOHh4ViyZAlMJpODb5+IiIjkzKGgM3jwYKxevRonTpzAiRMn8PDDD+ORRx6RwsyaNWuwfv16bNy4EcePH4darcaUKVNQV1cnPUdiYiJ27dqF5ORkZGRkoL6+HjNnzkRra6tUEx8fj7y8PKSkpCAlJQV5eXlISEiQpre2tmLGjBloaGhARkYGkpOTsWPHDixbtqy3nwcRERHJieil0NBQ8e6774q2tjahVqvF6tWrpWlNTU1CpVKJt956SwghRG1trfD39xfJyclSTXl5uejXr59ISUkRQghRWFgoAIjs7GypJisrSwAQ58+fF0IIsW/fPtGvXz9RXl4u1Wzfvl0olUqh1+vtbrterxcAHHoMEdHNYshvvhBDfvOF+OmaA55uCpEZR7bfTo/RaW1tRXJyMhoaGhAbG4vi4mJotVrExcVJNUqlEuPHj0dmZiYAIDc3F83NzWY1Go0G0dHRUk1WVhZUKhXGjBkj1YwdOxYqlcqsJjo6GhqNRqqZOnUqjEYjcnNzu22z0WiEwWAwuxEREZF8ORx08vPzccstt0CpVGLBggXYtWsXRo4cCa1WCwCIjIw0q4+MjJSmabVaBAQEIDQ01GZNREREl9eNiIgwq7F8ndDQUAQEBEg11iQlJUnjflQqFaKiohx890RERORLHA46w4cPR15eHrKzs/Hcc89h7ty5KCwslKYrLIbnCyG63GfJssZavTM1llasWAG9Xi/dSktLbbaLiIiIfJvDQScgIAB333037rvvPiQlJWH06NF44403oFarAaBLj0pVVZXU+6JWq2EymaDT6WzWVFZWdnnd6upqsxrL19HpdGhubu7S09OZUqmUjhjruBEREZF89fo8OkIIGI1GDB06FGq1GmlpadI0k8mE9PR0jBs3DgAQExMDf39/s5qKigoUFBRINbGxsdDr9cjJyZFqjh07Br1eb1ZTUFCAiooKqSY1NRVKpRIxMTG9fUtEREQkE/0dKX755Zcxffp0REVFoa6uDsnJyTh06BBSUlKgUCiQmJiIVatWYdiwYRg2bBhWrVqFgQMHIj4+HgCgUqkwb948LFu2DLfddhvCwsKwfPlyjBo1CpMnTwYAjBgxAtOmTcP8+fOxefNmAMCzzz6LmTNnYvjw4QCAuLg4jBw5EgkJCVi7di1qamqwfPlyzJ8/n700REREJHEo6FRWViIhIQEVFRVQqVS49957kZKSgilTpgAAXnzxRTQ2NmLhwoXQ6XQYM2YMUlNTERwcLD3Hhg0b0L9/f8yZMweNjY2YNGkStm7dCj8/P6lm27ZtWLJkiXR01uzZs7Fx40Zpup+fH/bu3YuFCxfigQceQGBgIOLj47Fu3bpefRhEREQkLwohhPB0IzzFYDBApVJBr9ezJ4iIyMJdL+0FAAy5bSDSfz3Rw60husGR7TevdUVERESyxaBDREQ28Zqe5MsYdIiIyKaezoVG5M0YdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIyCZe0pN8GYMOERERyRaDDhER2cYuHfJhDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkREZBMvdUW+jEGHiIiIZItBh4iIbFIo2KdDvotBh4iIiGSLQYeIiIhki0GHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZMuhoJOUlIT7778fwcHBiIiIwKOPPoqioiKzmmeeeQYKhcLsNnbsWLMao9GIxYsXIzw8HEFBQZg9ezbKysrManQ6HRISEqBSqaBSqZCQkIDa2lqzmpKSEsyaNQtBQUEIDw/HkiVLYDKZHHlLREREJGMOBZ309HQ8//zzyM7ORlpaGlpaWhAXF4eGhgazumnTpqGiokK67du3z2x6YmIidu3aheTkZGRkZKC+vh4zZ85Ea2urVBMfH4+8vDykpKQgJSUFeXl5SEhIkKa3trZixowZaGhoQEZGBpKTk7Fjxw4sW7bMmc+BiIiIZKi/I8UpKSlm/2/ZsgURERHIzc3FT3/6U+l+pVIJtVpt9Tn0ej3ee+89fPjhh5g8eTIA4KOPPkJUVBT279+PqVOn4ty5c0hJSUF2djbGjBkDAHjnnXcQGxuLoqIiDB8+HKmpqSgsLERpaSk0Gg0A4PXXX8czzzyDP/3pTwgJCXHkrRERUTd4pSvyZb0ao6PX6wEAYWFhZvcfOnQIERERuOeeezB//nxUVVVJ03Jzc9Hc3Iy4uDjpPo1Gg+joaGRmZgIAsrKyoFKppJADAGPHjoVKpTKriY6OlkIOAEydOhVGoxG5ubm9eVtEREQkEw716HQmhMDSpUvx4IMPIjo6Wrp/+vTp+O///m8MGTIExcXF+N3vfoeHH34Yubm5UCqV0Gq1CAgIQGhoqNnzRUZGQqvVAgC0Wi0iIiK6vGZERIRZTWRkpNn00NBQBAQESDWWjEYjjEaj9L/BYHDuzRMREZFPcDroLFq0CGfOnEFGRobZ/U888YT0d3R0NO677z4MGTIEe/fuxWOPPdbt8wkhoFDc6CDt/HdvajpLSkrCH/7wh+7fFBERddHNKpXIJzi162rx4sX47LPPcPDgQQwePNhm7aBBgzBkyBBcuHABAKBWq2EymaDT6czqqqqqpB4atVqNysrKLs9VXV1tVmPZc6PT6dDc3Nylp6fDihUroNfrpVtpaal9b5iIiIh8kkNBRwiBRYsWYefOnThw4ACGDh3a42OuXbuG0tJSDBo0CAAQExMDf39/pKWlSTUVFRUoKCjAuHHjAACxsbHQ6/XIycmRao4dOwa9Xm9WU1BQgIqKCqkmNTUVSqUSMTExVtuiVCoREhJidiMiIiL5cmjX1fPPP4+PP/4Ye/bsQXBwsNSjolKpEBgYiPr6eqxcuRKPP/44Bg0ahEuXLuHll19GeHg4fvazn0m18+bNw7Jly3DbbbchLCwMy5cvx6hRo6SjsEaMGIFp06Zh/vz52Lx5MwDg2WefxcyZMzF8+HAAQFxcHEaOHImEhASsXbsWNTU1WL58OebPn88AQ0RERAAc7NHZtGkT9Ho9JkyYgEGDBkm3Tz75BADg5+eH/Px8PPLII7jnnnswd+5c3HPPPcjKykJwcLD0PBs2bMCjjz6KOXPm4IEHHsDAgQPx+eefw8/PT6rZtm0bRo0ahbi4OMTFxeHee+/Fhx9+KE338/PD3r17MWDAADzwwAOYM2cOHn30Uaxbt663nwkRERHJhEIIITzdCE8xGAxQqVTQ6/XsBSIisnDXS3sBAPdE3oLUX433cGuIbnBk+81rXREREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBGRTQpev5x8GIMOERERyRaDDhEREckWgw4RERHJFoMOERHZpOAQHfJhDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6RERkk4KXLycfxqBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREQ28bzI5MsYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiMgmBUcjkw9j0CEiIiLZYtAhIiIi2WLQISIiItlyKOgkJSXh/vvvR3BwMCIiIvDoo4+iqKjIrEYIgZUrV0Kj0SAwMBATJkzA2bNnzWqMRiMWL16M8PBwBAUFYfbs2SgrKzOr0el0SEhIgEqlgkqlQkJCAmpra81qSkpKMGvWLAQFBSE8PBxLliyByWRy5C0RERGRjDkUdNLT0/H8888jOzsbaWlpaGlpQVxcHBoaGqSaNWvWYP369di4cSOOHz8OtVqNKVOmoK6uTqpJTEzErl27kJycjIyMDNTX12PmzJlobW2VauLj45GXl4eUlBSkpKQgLy8PCQkJ0vTW1lbMmDEDDQ0NyMjIQHJyMnbs2IFly5b15vMgIiIiORG9UFVVJQCI9PR0IYQQbW1tQq1Wi9WrV0s1TU1NQqVSibfeeksIIURtba3w9/cXycnJUk15ebno16+fSElJEUIIUVhYKACI7OxsqSYrK0sAEOfPnxdCCLFv3z7Rr18/UV5eLtVs375dKJVKodfr7Wq/Xq8XAOyuJyK6mQz5zRdiyG++EP/5xmFPN4XIjCPb716N0dHr9QCAsLAwAEBxcTG0Wi3i4uKkGqVSifHjxyMzMxMAkJubi+bmZrMajUaD6OhoqSYrKwsqlQpjxoyRasaOHQuVSmVWEx0dDY1GI9VMnToVRqMRubm5VttrNBphMBjMbkRERCRfTgcdIQSWLl2KBx98ENHR0QAArVYLAIiMjDSrjYyMlKZptVoEBAQgNDTUZk1ERESX14yIiDCrsXyd0NBQBAQESDWWkpKSpDE/KpUKUVFRjr5tIiIi8iFOB51FixbhzJkz2L59e5dpCouzSwkhutxnybLGWr0zNZ2tWLECer1eupWWltpsExER8YSB5NucCjqLFy/GZ599hoMHD2Lw4MHS/Wq1GgC69KhUVVVJvS9qtRomkwk6nc5mTWVlZZfXra6uNquxfB2dTofm5uYuPT0dlEolQkJCzG5EREQkXw4FHSEEFi1ahJ07d+LAgQMYOnSo2fShQ4dCrVYjLS1Nus9kMiE9PR3jxo0DAMTExMDf39+spqKiAgUFBVJNbGws9Ho9cnJypJpjx45Br9eb1RQUFKCiokKqSU1NhVKpRExMjCNvi4iIiGSqvyPFzz//PD7++GPs2bMHwcHBUo+KSqVCYGAgFAoFEhMTsWrVKgwbNgzDhg3DqlWrMHDgQMTHx0u18+bNw7Jly3DbbbchLCwMy5cvx6hRozB58mQAwIgRIzBt2jTMnz8fmzdvBgA8++yzmDlzJoYPHw4AiIuLw8iRI5GQkIC1a9eipqYGy5cvx/z589lTQ0RERAAcDDqbNm0CAEyYMMHs/i1btuCZZ54BALz44otobGzEwoULodPpMGbMGKSmpiI4OFiq37BhA/r37485c+agsbERkyZNwtatW+Hn5yfVbNu2DUuWLJGOzpo9ezY2btwoTffz88PevXuxcOFCPPDAAwgMDER8fDzWrVvn0AdARERE8qUQQghPN8JTDAYDVCoV9Ho9e4GIiCzc9dJeAMCoO1T4fPGDHm4N0Q2ObL95rSsiIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIisolXLydfxqBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERDbxoCvyZQw6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkREZBsvX04+jEGHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZItBh4iIiGSLQYeIiIhki0GHiIhs4sHl5MsYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIyCZevJx8GYMOERERyRaDDhEREckWgw4RERHJFoMOERERyRaDDhEREcmWw0Hn8OHDmDVrFjQaDRQKBXbv3m02/ZlnnoFCoTC7jR071qzGaDRi8eLFCA8PR1BQEGbPno2ysjKzGp1Oh4SEBKhUKqhUKiQkJKC2ttaspqSkBLNmzUJQUBDCw8OxZMkSmEwmR98SERERyZTDQaehoQGjR4/Gxo0bu62ZNm0aKioqpNu+ffvMpicmJmLXrl1ITk5GRkYG6uvrMXPmTLS2tko18fHxyMvLQ0pKClJSUpCXl4eEhARpemtrK2bMmIGGhgZkZGQgOTkZO3bswLJlyxx9S0REZAOPLidf1t/RB0yfPh3Tp0+3WaNUKqFWq61O0+v1eO+99/Dhhx9i8uTJAICPPvoIUVFR2L9/P6ZOnYpz584hJSUF2dnZGDNmDADgnXfeQWxsLIqKijB8+HCkpqaisLAQpaWl0Gg0AIDXX38dzzzzDP70pz8hJCTE0bdGREREMuOWMTqHDh1CREQE7rnnHsyfPx9VVVXStNzcXDQ3NyMuLk66T6PRIDo6GpmZmQCArKwsqFQqKeQAwNixY6FSqcxqoqOjpZADAFOnToXRaERubq7VdhmNRhgMBrMbERERyZfLg8706dOxbds2HDhwAK+//jqOHz+Ohx9+GEajEQCg1WoREBCA0NBQs8dFRkZCq9VKNREREV2eOyIiwqwmMjLSbHpoaCgCAgKkGktJSUnSmB+VSoWoqKhev18iIiLyXg7vuurJE088If0dHR2N++67D0OGDMHevXvx2GOPdfs4IQQUnc4zrrByznFnajpbsWIFli5dKv1vMBgYdoiIiGTM7YeXDxo0CEOGDMGFCxcAAGq1GiaTCTqdzqyuqqpK6qFRq9WorKzs8lzV1dVmNZY9NzqdDs3NzV16ejoolUqEhISY3YiIiEi+3B50rl27htLSUgwaNAgAEBMTA39/f6SlpUk1FRUVKCgowLhx4wAAsbGx0Ov1yMnJkWqOHTsGvV5vVlNQUICKigqpJjU1FUqlEjExMe5+W0REROQDHN51VV9fj4sXL0r/FxcXIy8vD2FhYQgLC8PKlSvx+OOPY9CgQbh06RJefvllhIeH42c/+xkAQKVSYd68eVi2bBluu+02hIWFYfny5Rg1apR0FNaIESMwbdo0zJ8/H5s3bwYAPPvss5g5cyaGDx8OAIiLi8PIkSORkJCAtWvXoqamBsuXL8f8+fPZU0NE5ELdDQcg8gUOB50TJ05g4sSJ0v8dY17mzp2LTZs2IT8/Hx988AFqa2sxaNAgTJw4EZ988gmCg4Olx2zYsAH9+/fHnDlz0NjYiEmTJmHr1q3w8/OTarZt24YlS5ZIR2fNnj3b7Nw9fn5+2Lt3LxYuXIgHHngAgYGBiI+Px7p16xz/FIiIiEiWFEII4elGeIrBYIBKpYJer2cvEBGRhbte2gsAiBkSih3PjfNwa4hucGT7zWtdERERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQEZFNPIsO+TIGHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiKyiRcvJ1/GoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREQ2KXj9cvJhDDpEREQkWww6REREJFsMOkRERCRbDDpEREQkWww6REREJFsMOkREZBsPuiIfxqBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERESyxaBDREREssWgQ0RERLLFoENERDbxfIHkyxh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIyCYFRyOTD2PQISIiItlyOOgcPnwYs2bNgkajgUKhwO7du82mCyGwcuVKaDQaBAYGYsKECTh79qxZjdFoxOLFixEeHo6goCDMnj0bZWVlZjU6nQ4JCQlQqVRQqVRISEhAbW2tWU1JSQlmzZqFoKAghIeHY8mSJTCZTI6+JSIiIpIph4NOQ0MDRo8ejY0bN1qdvmbNGqxfvx4bN27E8ePHoVarMWXKFNTV1Uk1iYmJ2LVrF5KTk5GRkYH6+nrMnDkTra2tUk18fDzy8vKQkpKClJQU5OXlISEhQZre2tqKGTNmoKGhARkZGUhOTsaOHTuwbNkyR98SERERyZXoBQBi165d0v9tbW1CrVaL1atXS/c1NTUJlUol3nrrLSGEELW1tcLf318kJydLNeXl5aJfv34iJSVFCCFEYWGhACCys7OlmqysLAFAnD9/XgghxL59+0S/fv1EeXm5VLN9+3ahVCqFXq+3q/16vV4AsLueiOhmMuQ3X4ghv/lCPLE509NN6TPF1fXiXydKRUtrm6ebQjY4sv126Rid4uJiaLVaxMXFSfcplUqMHz8emZmZAIDc3Fw0Nzeb1Wg0GkRHR0s1WVlZUKlUGDNmjFQzduxYqFQqs5ro6GhoNBqpZurUqTAajcjNzbXaPqPRCIPBYHYjIiLqMGHdISz79DT+eaLU000hF3Fp0NFqtQCAyMhIs/sjIyOlaVqtFgEBAQgNDbVZExER0eX5IyIizGosXyc0NBQBAQFSjaWkpCRpzI9KpUJUVJQT75KIiOTu+KUaTzeBXMQtR10pLI5FFEJ0uc+SZY21emdqOluxYgX0er10Ky1lYiciIpIzlwYdtVoNAF16VKqqqqTeF7VaDZPJBJ1OZ7OmsrKyy/NXV1eb1Vi+jk6nQ3Nzc5eeng5KpRIhISFmNyIiIpIvlwadoUOHQq1WIy0tTbrPZDIhPT0d48aNAwDExMTA39/frKaiogIFBQVSTWxsLPR6PXJycqSaY8eOQa/Xm9UUFBSgoqJCqklNTYVSqURMTIwr3xYRERH5qP6OPqC+vh4XL16U/i8uLkZeXh7CwsJw5513IjExEatWrcKwYcMwbNgwrFq1CgMHDkR8fDwAQKVSYd68eVi2bBluu+02hIWFYfny5Rg1ahQmT54MABgxYgSmTZuG+fPnY/PmzQCAZ599FjNnzsTw4cMBAHFxcRg5ciQSEhKwdu1a1NTUYPny5Zg/fz57aoiIXEgBnhqZfJfDQefEiROYOHGi9P/SpUsBAHPnzsXWrVvx4osvorGxEQsXLoROp8OYMWOQmpqK4OBg6TEbNmxA//79MWfOHDQ2NmLSpEnYunUr/Pz8pJpt27ZhyZIl0tFZs2fPNjt3j5+fH/bu3YuFCxfigQceQGBgIOLj47Fu3TrHPwUiIiKSJYUQQni6EZ5iMBigUqmg1+vZC0REZOGul/YCAGJ/cBu2PzvWw63pGx3v+bH/uAPr5/zYs42hbjmy/ea1roiIiEi2GHSIiMimm/Lq5Tftvg75YdAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIiLZYtAhIiIi2WLQISIiItli0CEiIrLA8wXKB4MOERHZdFOeGZlkg0GHiIiIZItBp49p9U1YsTMfRdo6m3VXahuxYmc+LlTarrNGCAFDU7OzTeyi9roJr+zKx6kSnXTfV2e1+NHvU/Duke+sPqa5tQ0rPzuLg+erun3etjaBN/ZfwKEi2zWr9p3DvvwK59+AC/0rtwwb0r5x+vE1DSakf1ONtjbbHeN1Tc1I/6Yaza1tZvfvySvHmpTzEKL3HeuOPNdfv76APXnlDr+GVt+E3+62bzn++lwl/vD5WbRYvOfeOPxNNZ58Owsrdp5BdZ0RF6vq8Nvd+ajQN6K6zohfvH8cSz/Jw68/PY0Tl2psPtd1Uwt+v6cAWd9eA9D+PXv+45N4eN0h6fuW9e01/G53Aa6bWuxuY+7lGry6pwB1LvzO9uTTE6W466W92H2qHL/bXYB64432phRo8dKO9s+rw9GL1+x63s9OX8HfD17scv+31fV4L6MYTc2tvW/890quXcfjmzLx5NtZqNA3SveX1zbit7vz8V11PbbnlJi1Z3tOCd5K/9ZlbQCA//uiEHe9tBdluus91ra2CaxJOd/tOi/9m2qs/OwsjC2OfU6ZF6/i93sK0Ghy3ecLAAXlery8K99sWfBF/T3dgJvN8x+fRO5lHf55ohTfrvpP6f4KfSN+v+cs0gor8cy4u3CyRIczZXrsOlWG8/833ew5UgoqsOCjk5g1WoNfTR6GH9x+C4QQePdIMX58563YdaocHx8rwcfzx8DY3IbDF6rx8weGIipsIID2L9uRC9X4cdStuHVgAEwtbWgwtiA0KMBqm1/7vBA7T5Vj27ESHHt5EiJDBuCXH+YCAP649xz+K2Ywbh1447HX6o1Y+XkhPj99BVszL+HS6hkAgLzSWoQO9MeQ24IAtIelDfvbQ0NHjaXUQi3ePvydzRpXOlWiwzeVdZhzXxQUCgXOXtFD39iMcf8WDgBY/ulpAMCkERH4kUaFBlMLQgb4W32uIxeqcey7Gvxqyj3w69fe9x+3IR1X60348+Oj8MT9dwJo32CeLKnFsMhbpOea+48cnCypxZJJw7B0yj3Sc76QnAcA+MHtt+C/Ygb36r12PNcDd4fjgbvDrdYIIbDo41PY+33QfOTHd0jTGowt8OunwAB/v25fY9HHJ3Hisg7/PFGGb/54YzlubRPYnlOC0YNvxYhBwejv1w/z3j8BALgnMhhP/eROp95TS2sb/PopoPh+X8v/+0cOACD7uxoc/uYqaq+b0GBqRUG5AeG3BGD/uRsbnE9zy6Rl7ExZLfz9+mHEoBBp+sYDF/FB1mV8kHUZl1bPQNZ317D3TPvnsu6rIrz2SDSeeicbAHDLgP74zbQf2tXmxzdlSX//4ZFos2mNplY0t7UhZIA/rtYbkZichzn3R2F6tBr+frZ/pzYYW5D57TU8NCy8yzz69b/OAAASP8kDAPj79cOj/67Bhcp6LPt+Ga+y2Lg1NbfanNcAsGT7KQDAg3eHY3TUrdL9k15PBwDoG5vNlufeeGZLDr672gAAWPzxKfzruXEAgF+8fwLnKgzYl69FTYMJAPCfowZhaHgQVuzMBwDMGDVIWh+aWtoQ0N++3/ymljY0NrdCFegv/f9eRjEA4NG/H8WJ306x+fjdp8rx5qFv8eahb62uz+Z+v7xqbh2AZ3/6b3a1CQDi3z0GALh1YECXz7e05jpCgwJwi9Lxzf3Mv2UAACr1TXjvmfsdfry3YNBxowZjC+qaWqBWDZDuK7xiANC+ov/s9BVphfXiv87gyIWrAICtmZfw/XYRTc3tv25rr5vw/Mcn8di/D5ZWRJ+fvoLPT18BADz1kyhszyk1e/34d45Jf285eiNwvJ95Ca99UYg7wwbi8IsTMWHtQVzRNyF7xSSztna4UFUv/T1m1dc499o0s+nGFvNf4A/++SAaLX65ldZcx6N/PwrgRmApr21ET5z5JbHp0Lf4sqAC234xBsHdhJDu/OzNTACA5tZAPDTsdsz4a/sXPfOlh6G5NVCq233qCmZvbH8/R16ciKiwgSituY7f7SnAvAeH4o5bA5HwXvtKa+PBi/jnL2Pxk6FhuFrfvuL96mwlnrj/TjSaWhG7+mvUXm//NT9x+O34xzP342RJLQDgn8dLsXTKPWhrM++lW/7pafxXzGAYW1rR2iYwMKD7r3JTcyt+v6cA9w0Jw5z7o7pM79gYWJP+TbUUcjprNLXiR69+hVuU/VHwh6ndPr7gih5A+wahs+TjJfjt7gIAwNDwIBxcPkGaVqFv6vb5utPc2r7BWf3lecwercFfn/r3LjWdl7e80tpun8vQ1CzN21F3qLA07h5MHB4hBW6gvTes83L/jUWPVUlN97/uT5bosOdUOZZNHW4Wkt/Putwl6Ix+LRWmljYU/GEq/vzleWRcvIqMi1cRMqA/cl6ZbDN4PLftJA5/U42nfnInkh4b1W0dAPzjaDH+cbTY7D7LXueWHnohO/v9ngLsWfRgl/tPXtZZqXZOR8gBgHMVhi5/d16u65ta8GWn5bjh+x63HbllWPbpaWx4YjR+9u89/3DoWFee+O1khN+ixLZjl6VpHd9tW67Ysc4DgHJd17q2NoHET/IwXB2M5yfebfVxp0p0OFNWi3sH3wqgvSetI2Sm/3qC9CPTUUUO7llIKdDiizNX8Jcnfoz+PQTyvuD5FsjYfX/cj7FJX5st3J0DwJLtpzDslS9xpbbR5ooRAN74+gKOXrwmhRxLliHHmuzvruFCZR1e+6IQQPvK+K30b3Hl+w3L4W+qe3wOoOfwYRlyAOBCleO74Jz155TzOFOmx5ajl5x+jm87hTsAZl3jAMw2CnO3tAeaX32Sh0NF1Uh4LwcPf79y6fDzrce7vIahqRkjfp8ihRwAOFhUjfRO80FraJ83P3//OH78WlqX57j/j/sx8vdf2dwlsOXoJfzzRBle3HEGH2Vf7rbOmlIrK1ygfQUKwGyXhyPyvg9yAFDcaYNlL/118908W462hxygffdJWmElEt47Zu2hParptMHKL9fjf7e0z7vOG/qO3rAO2d/VSJ9JTx57MxPvZ12W2mtLR0C8UFkHXaf3bGhqwalOn6E1Hd/n7TkldrXLlU6X6aF1IrDay5nl7rltJ7vc17E+/dUn1terlq/Zsa6c9X1Pxx8+L3S4HR0OnK/Em4cu2r0bOvu7a/js9BWs/aqo25ojF65i9sajKChv/4FxpNO6ZPzaQ0631VELPsrFF2cqcPcrX/bZa9rCoONGHRv84z3s+7e2EbRkaHRug9LZk29n4+fvm7+WPStb0cOBljUNJhwqqnLJuBFXsuxFcJfvqts31LZ6IixXzEIIHPnmqtVaQ1PXeX2oyHoI7ai1FRYqDTfa1dGL0tmFKvs20M5wxyKR9OU5jH4tFU9szsKZsloAkMbNdJj/wQmph9QRvVmGL1R2/RxbbfSCfGdnMHIFR8YMuYo7v3/2jIfpiWUvXE86j/VxptfR0s+3nsCalCK7xz9Z+wHZHVvjzVrbBHIv6xyaP735Hse/k42LffhD1xoGHS9wvoeBya50pdbxL2hPC/n0N47gmS3H8eCfDzrZKnKnNiszsPb6jV6Lv359oS+b02ub09t3IR0rrpF2MbmKQyv0HmozLlzFiN+n4NMTPfe2utvD69J7LuoDPf1o6iv6682I23DYocdU2LHbaevRYjz3UW6Xgwhs0Rrc1/NlzZqvzuPxTZn49b967sWyV8aFq9KuQcuevMxvr+EX34+/8xQGHTdoaxN27wbq4GWdIU4pr2106REzNyujnb/c0gore6wRQuCDrK67qx7qZSi1tbxWGZow560s7Mkr95LNmmOstdmZ5foXHxyHqaVNGvjrSa7YmHpbj21v2NMj48z7Xfl5Ib4s0OKZ73dnWzr6reO9jK7W8UNhT94Vlz3n/7x3DM9tO4kKfSPePNT1qLu+DnOWGHTc4OOcEuloD6B9f36VgzNaYXGGLl85YZezYzZ8naOHgwp0/+vW3vXr/A96/pXU0M3hpnUW86m6zmj9lAdOrOz/uPccci7VdBnH4stetCus3Pis7Pm6entusOdgAW/hiY/SVhA6evEajl7sGmqyv7M9jKGD5frfV3R3YEPHQTWewqDjBl8WdD1K5dXPznqgJV1Z243RE3etkL19Re+Iv359AVfr7T9C7FBRNVLP9twjY6/efpb3/2k/pv7lsN0Dg23tgtA3Nncu7Obx3kmhsL4B23nK8XMIuZIvbvjc2WSFlShZZWjCwm257ntRCx1HaHbnosXYt9IeDjjpiTs+z89OX0Gyk4PVqwxN+M83jpgdeQZ453qdQaePVPbQo9NX+65tLoSdvkjGllYk7TuH7O+6DpRz5gtnbcXkbToPzrPnUNHO9hdWdTnMviefnbbedezMsuCq5eers1qXPE9vZH9r3+BMd8n//oiVntj6zAXMv2sr3fRDx9DU3GeD7l0Rtty5EXx5Vz725Vtffm2dlNRelk23dYoCS1p9Ex5a431jGJdsP4WXduY7vMcBANZ+VYTCCgNe2dX1AAdvw6DjI/o6Jmw5egmbD3+HJ9/OdlsE68uBieW1jdIhl93p3Nt13WKXj6GxBW8fdu0ZVfuCo+MMOk5+JoRAybXrvZ5DzszjnEs1fXpEUmf78rVu2eW2NfNSl/sc/WQsc4a+0YR7V6bioTUHnG6XXFw3taKsm1MhAO3LVGd93UF2rNix8N7X46Esd2Xb47qVsYTe+oOWJwz0Et7W3Xf52o1dGHIYhPjA6vaNweFfT8Sdtw3ssd5yRbhiZ77NAXWOnlDLFlOr5z/vvx24iPU2LnVh7yLh7KJTfLUBP7j9FrtqO58srrd2nSpz2XP1SNg+/LwnJy61n3yv0tA3p+eXw3qAnOPIvPeWI+s6Y4+Ol/LOXOz7Cu3cKFp+r/vyqIHfWTnXTV/rLuQ0Nbdi69FiXLrm+En+3GX6G0c83QQA5suMAj332ORcqsH9f9pvdt+1eiOe2JzVJXD54vrArWN0fPED6SVP9pYIwO7rlHljHmbQcQNrC2RP+7e9ceEg17I8m68rdSw/f/36Aqa/cUS6XISrF6uNBy5i5eeFZrt3dp0qk82vfUfehivesuVRKmu/KsKx4hq7ztTr7ayt87xlMemLwd2dX6LFRi+tte+Otw0+r9A34Ye/S3HogAtvwqDjI7xpuXe0LYamZvyvHWd/lrvRr6W6/TXWp32DcxUGvO/k5S96mrU5xV0Pj/3VJ6dxwMZV6p1qh4eWd09vh82OWOvE0+3yBa4OB60C+PWnp6XLpvQmpH3sgctwuMOXVq5715lC4Z3LKoOOl7A8Z4UjF9Bzjxsrjd7+Cut86nRf4Y37mR3R7OTy4+y7PnvF+i7B7p7PW37ZW3Jpz5SXvsfeMDQ1458nSt3WO3mhsg7vZRS7/UgyeyJRWqEWn+aWWb1siqOv0XGpEqt19ga0TmUvJJ/Cgg9zZdOT6m4MOn3EizpkumXfSdEcZ+juV6qbv6OOhhVv6jXrrW+r6rH7VLnLP2NHP9PerIhPlugwZX26XWcZ95UTVTrbTm9YNBUKBZZ+kocX/3UGz7npfDVTNhzG/31RKB3950mePsmdLXvyriDlrNbjZxz2FQw6buCODaa3HrYnV1U9XKHd2+3Nr0DiJ3l9dl4cVwcqBRT4n3eP4UJVvdlZxrtz/PsjkPqS5Vvu3InW3cDNzem2T1Fg63P09BpACIH959p3UWbaca4ja+21Nyj/OeW81XN42XpuR3j7jxprpyOwpq87/nt6ufYTbvZJUxzCoENWdV4R+PpunM66W8GV6a5j4Ucnpf/3nrG9L7ov2dONLwSQtO9cl/vtuTq9pUUfn+x2WkF5z0etVehv7IbtzYrY8lxGfaE3S3pL2435pDU0Wf3edHeKfF/wxn7HLv7a2zDx5NvZXe5rNLXiF+8fxz+94EKpPfL2NOUE+0KM920vGHT6iC9dNwYw/8VkuXD36REBffRaicl5+NpiQG2Wh8/Q22HL0Z678TMuXsXmw991ud/RDWt1nRFf2Ah5jXYcYvpNZc8n++sxPHtqMLKH19EnLtt3LSRPeNdid9Lfvr7Qq/MAOWNr5iXsP1eFd450/U44ssgcOOfawfN9wdr7W/7P0z4dnvsKg04fsedqubb49o8Da4eZelc/UYmV69B0d4mGvnahqufg0LkXpSdVda7fr+/quempxd3Ty2R3lx7JKa5BqsXV6h1dJ1TXGZGYfAonLrkmTL2e9g12nmw/309bHwWe7o5Kc5Qz1y7rzTvsaehBpaHJoUtKdMj67hpe+7zvrqPYeczd1+e6XqtPAYXHfyxYwzMjU49cvdwKIfDUO9l2X8m3w4XKOnx++grm//QHCB7g7+JW+bYPsi73XPS9V/d4wQVmnViortQ2Ys7mLNe3xU0sV/hHL151+rv0Jyu7JR31yq58pBZWYnee6wL8pWsNOHGpBv/vHzn47YyRPdZ740bQXRTd/tPVmFVfO/061n6kuUvH7GtqbrU6WHv1l+eQYeWq7Z7GoEMO6+2v7Tpji8MhB2g/IgNoHyi8+vF7e36dphac1xrwQ3VIj7V93QXvSb3tXewL1s5gvfarIpvXM+prPW20LSc//e4x3KL03CrXHRvE1LOV2JFbjuumVry8K99smjsOoHBnP3B5bSO2Zdv/g8FVfPEQ8e4uYHywqOcjJD2BQcdHeHLXlbd9Ee3t4v0g6zI+yLqMTxfEdpnW0toGv34KabzRtZtoP7cnD8UWQuC8tg6mVtsDrNekFJn9f7XeiF1O7G5wVG+WdXvGrrnys/eGr+WFqnqoQwZYneaWdZato9J6+XpPv5ONS9f6rnfEUa4YG9nSw/euJ96wzDmDQccNvO303c4wP+qq+2m+YH+nfckKAA3GFjy05iBGD1Zhy//+xHMN85CLdoz5cVTHCvDTHo6G2Z5T2uWXvz06HxHnrSxDkrf9QPBlJy7VICrM9sV4e/tx9xRyvqt27HvTeT3pzlWmI73Rd7/yZa9eq+OVfG0bwKDTh3zpyCtXdjt725fiUFE1ahpMXtvN6quOX6rBr3s46eTWTOdOBJfjogG0PXFVNmlp9a7B9u7kyEnrnPlMci/X4L/eah+b9exPf+DEM7jGeW2dW57Xnh/GQgg0dNMbeLpM7+omyQ6Djht0t9hqfWBshNxV1hkRNjDA081wiK90DBRX93xFc195L/awjDKd35u7Noo3I2fG88nN8k/PYMfJsp4L3cxXeykZdHyGJwfpeO6lXe13uwtwd8Qtnm6GQ7xhBdeT3i4ip504tNYdvPGIEV9lbY1V3+T4GCVv2bg62o69ZyqgUQVi4g8jet2r7cw6oKBcD32je8bjeVknfY8YdMgqW2N0fJ07xqhQ7zzy96OeboLX6WnD6m27hLuw0j5rR9P1pPPHYOszcfXnYXk5A0fXg5nfXkPmt9dwafUMp9uwJuU8jjpx4tJjxTVY+Xmh068rNy4/YeDKlSuhUCjMbmq1WpouhMDKlSuh0WgQGBiICRMm4OxZ8/N6GI1GLF68GOHh4QgKCsLs2bNRVmaeaHU6HRISEqBSqaBSqZCQkIDa2lpXvx2ygtfdImvsGZUit9Dcmas3tPZcT+pm0HmZ8WTnTm9e29l15puHvnWqt/NrB8/8bG/r1n5VhF2nynzugBu3nBn5Rz/6ESoqKqRbfv6NoyzWrFmD9evXY+PGjTh+/DjUajWmTJmCurob+7QTExOxa9cuJCcnIyMjA/X19Zg5cyZaW2+cfj4+Ph55eXlISUlBSkoK8vLykJCQ4I6347DulgFnznwJAMk5Jb0+LNBRuZddd5FE3/pKkFO8ZPdCX3L3W2bPYztPLVo343rr63OVWP7paVw3Wd/lZWxpw68+Od3Hreo9t+y66t+/v1kvTgchBP7yl7/glVdewWOPPQYAeP/99xEZGYmPP/4Yv/zlL6HX6/Hee+/hww8/xOTJkwEAH330EaKiorB//35MnToV586dQ0pKCrKzszFmzBgAwDvvvIPY2FgUFRVh+PDh7nhbvXbeiW5bAHhpp+OH4/bW2Ss32mrZXewtYb7K0AQBILKb83j0lre8T1/w1wMX8cvxPR8R4y3jLdyhr9+ao6/X1wOkFVDgTFktvrNjkLq9+vIjVvTRpbjd8Z1w9pi/ee+fAADccWsgfjXlnm7rfG3V6JYenQsXLkCj0WDo0KF48skn8d137RcbLC4uhlarRVxcnFSrVCoxfvx4ZGZmAgByc3PR3NxsVqPRaBAdHS3VZGVlQaVSSSEHAMaOHQuVSiXVWGM0GmEwGMxufclXV/He2O7m1jb8ZNXXGLPqaxhb+v4q19TV5vSuFxW9mdhzlXlX+vyMd1yLzZbZG48i8ZO8Xj2HvRttWz9MnNnV4umzpe/L7/7iuu7mjuvheZLLg86YMWPwwQcf4KuvvsI777wDrVaLcePG4dq1a9BqtQCAyMhIs8dERkZK07RaLQICAhAaGmqzJiIiostrR0RESDXWJCUlSWN6VCoVoqKievVeHdXXu5+c8dVZLZ7/2PmTs/XVr9rO55To6UJ/HFPkPbwxNLvKiztsn0PI1SoNxj59PUe56kxC5oORXfKUfapMdx2Nzd3/GLtusj5t4Tbn18PNre79oI5c8K2jE12+62r69OnS36NGjUJsbCz+7d/+De+//z7Gjh0LoGu6FkL0mLgta6zV9/Q8K1aswNKlS6X/DQaDW8JOdy1w5cX03OWXH+Z2ua+3Kxe37wLywZUf+T4udrZ9md/9j05HmA1GdvJT9+Qu00c22j6i8NXPvOAiuxa+PleFi1Xd7+rszY9hT3DLrqvOgoKCMGrUKFy4cEEat2PZ61JVVSX18qjVaphMJuh0Ops1lZVdLxFfXV3dpbeoM6VSiZCQELMbOa63uaW6zv5fotZeSwjzXhpucHwIZ5bdfH2M2DknxyR20SmkGNx0Xhh38sXr6FXVGTF5/eFup3t6t56j3B50jEYjzp07h0GDBmHo0KFQq9VIS0uTpptMJqSnp2PcuHEAgJiYGPj7+5vVVFRUoKCgQKqJjY2FXq9HTk6OVHPs2DHo9XqpxpN6umChr3H1yeyf2Jzd6+fI+u5G16m7fqz5Yjc5kbdIPm77umcdetpodp7qqZNnynkQ/c3A5buuli9fjlmzZuHOO+9EVVUV/vjHP8JgMGDu3LlQKBRITEzEqlWrMGzYMAwbNgyrVq3CwIEDER8fDwBQqVSYN28eli1bhttuuw1hYWFYvnw5Ro0aJR2FNWLECEybNg3z58/H5s2bAQDPPvssZs6c6RVHXB29yPNf2HK1vndjC0p117Gg00UeOwex3j43uRc3F+5lbGnFku2n8OCw2z3dFLvc9dJeAMCO58YhZkhol+nXTS129x7YGovX2/O+fGJnaCPv5PKgU1ZWhqeeegpXr17F7bffjrFjxyI7OxtDhgwBALz44otobGzEwoULodPpMGbMGKSmpiI4OFh6jg0bNqB///6YM2cOGhsbMWnSJGzduhV+fn5SzbZt27BkyRLp6KzZs2dj48aNrn47BO/r2bAcvNe5fav2nevj1tBNy81fDGeefkduOb46W4mvznbdte9t9uSVS38/vikTowerutSM/P1Xfdmkbn2a6/2XYaHuuTzoJCcn25yuUCiwcuVKrFy5stuaAQMG4G9/+xv+9re/dVsTFhaGjz76yNlmUm84+OOoL496ulrve/vDbyZy2gXQ1Ox9u6jrmmwfgehNfru7wOz/3l6F253Xi2/xsTEpZM7tY3TI95XpGs3+97bDtc1PEc8VkrcqrbmOUotlyZdtPuze8wY5urfluqkFSV+ed09j3MC71iK2ectFZ8k5DDrkFd48dNHpkfwMN77hoTUHfe5oDVvcPR7M0cV69ynvP30FcKPXydXXS7L1A+zwN9UufS3yLQw65BXWpBTh0xPODfjr2CDUNJhQoe96Rk9n16e+fngv+bYmGyeZs/Ta54VobfO+XWnW/H5P+3lj+P2ivsKgQ17j2+reXcTwP/4vjRdCJNlwZDfUP44W+8wRbbtOtQ9CdnXOqfHB89VQ32DQIbf76qz9Z0gVQiClQIvL1+y/EGBPXfy2ptde58qRyBNcveuqvFY+47/ItRh0yGGOrp+q7DwTskKhwNfnqrDgo1yMX3vI7ufvzdEWS5LznH4skTfhUDXqjdKa655ugtsw6JBXyS3R9VxkoTcreA5SJLnYduyyp5vgEO5q8h7ltY14aM1BTzfDbRh0yGFtbjpyxtmObHf9kK20MrCZyFt9U+k749Oq6vjd8iZvu/lUCZ7GoEMOO1Zc454ndjLpuOvw8q/PV7nleYludrFJBzzdBOrkyIWrPRf5MAYdcliLmw5jtTwPxuenr+BCZZ1bXouIPEdO51Mi7+fyS0DQzW3BR7lOP1ahMB9vs3j7Kbsex1UmERF1hz065DBbe4rctlvLhp72XPV0lNhvd+e7rjFERORVGHTIqzS3OrNbzHbS2XToW5vTP8ouceI1iYjIF3DXFTnMXefr6CmQdMepbERERDcF9uiQz/vYx84fQkREfYdBhxxW5GVHQv0rt8zTTSAiIi/FoEMO+zDLu3pQeNQVERF1h0GHHNbY3OrpJphp40V+iIioGww65POYc4iIqDsMOuTzjC1tWPdVkaebQUREXohBh2Rh48GLnm4CERF5IQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItBh0iIiKSLQYdIiIiki0GHSIiIpItnw86b775JoYOHYoBAwYgJiYGR44c8XSTiIiIyEv4dND55JNPkJiYiFdeeQWnTp3CQw89hOnTp6OkpMTTTSMiIiIv4NNBZ/369Zg3bx5+8YtfYMSIEfjLX/6CqKgobNq0ydNNIyIiIi/gs0HHZDIhNzcXcXFxZvfHxcUhMzPT6mOMRiMMBoPZjYiIiOTLZ4PO1atX0draisjISLP7IyMjodVqrT4mKSkJKpVKukVFRbmlbYNDA93yvEREROQYnw06HRQKhdn/Qogu93VYsWIF9Hq9dCstLXVLmw4un+CW5yUiIvI1+5Y85NHX7+/RV++F8PBw+Pn5dem9qaqq6tLL00GpVEKpVLq9bf5+/XBp9Qy3vw4RERHZ5rM9OgEBAYiJiUFaWprZ/WlpaRg3bpyHWkVERETexGd7dABg6dKlSEhIwH333YfY2Fi8/fbbKCkpwYIFCzzdNCIiIvICPh10nnjiCVy7dg2vvfYaKioqEB0djX379mHIkCGebhoRERF5AYUQQni6EZ5iMBigUqmg1+sREhLi6eYQERGRHRzZfvvsGB0iIiKinjDoEBERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFsMegQERGRbDHoEBERkWwx6BAREZFs+fQlIHqr46TQBoPBwy0hIiIie3Vst+25uMNNHXTq6uoAAFFRUR5uCRERETmqrq4OKpXKZs1Nfa2rtrY2XLlyBcHBwVAoFC59boPBgKioKJSWlvI6Wl6G88a7cf54N84f73UzzRshBOrq6qDRaNCvn+1RODd1j06/fv0wePBgt75GSEiI7Bc4X8V54904f7wb54/3ulnmTU89OR04GJmIiIhki0GHiIiIZItBx02USiVeffVVKJVKTzeFLHDeeDfOH+/G+eO9OG+su6kHIxMREZG8sUeHiIiIZItBh4iIiGSLQYeIiIhki0GHiIiIZItBxw3efPNNDB06FAMGDEBMTAyOHDni6Sb5lMOHD2PWrFnQaDRQKBTYvXu32XQhBFauXAmNRoPAwEBMmDABZ8+eNasxGo1YvHgxwsPDERQUhNmzZ6OsrMysRqfTISEhASqVCiqVCgkJCaitrTWrKSkpwaxZsxAUFITw8HAsWbIEJpPJrCY/Px/jx49HYGAg7rjjDrz22mt2XX/FFyUlJeH+++9HcHAwIiIi8Oijj6KoqMishvPHczZt2oR7771XOmFcbGwsvvzyS2k65433SEpKgkKhQGJionQf54+bCHKp5ORk4e/vL9555x1RWFgoXnjhBREUFCQuX77s6ab5jH379olXXnlF7NixQwAQu3btMpu+evVqERwcLHbs2CHy8/PFE088IQYNGiQMBoNUs2DBAnHHHXeItLQ0cfLkSTFx4kQxevRo0dLSItVMmzZNREdHi8zMTJGZmSmio6PFzJkzpektLS0iOjpaTJw4UZw8eVKkpaUJjUYjFi1aJNXo9XoRGRkpnnzySZGfny927NghgoODxbp169z3AXnQ1KlTxZYtW0RBQYHIy8sTM2bMEHfeeaeor6+Xajh/POezzz4Te/fuFUVFRaKoqEi8/PLLwt/fXxQUFAghOG+8RU5OjrjrrrvEvffeK1544QXpfs4f92DQcbGf/OQnYsGCBWb3/fCHPxQvvfSSh1rk2yyDTltbm1Cr1WL16tXSfU1NTUKlUom33npLCCFEbW2t8Pf3F8nJyVJNeXm56Nevn0hJSRFCCFFYWCgAiOzsbKkmKytLABDnz58XQrQHrn79+ony8nKpZvv27UKpVAq9Xi+EEOLNN98UKpVKNDU1STVJSUlCo9GItrY2F34S3qmqqkoAEOnp6UIIzh9vFBoaKt59913OGy9RV1cnhg0bJtLS0sT48eOloMP54z7cdeVCJpMJubm5iIuLM7s/Li4OmZmZHmqVvBQXF0Or1Zp9xkqlEuPHj5c+49zcXDQ3N5vVaDQaREdHSzVZWVlQqVQYM2aMVDN27FioVCqzmujoaGg0Gqlm6tSpMBqNyM3NlWrGjx9vdoKuqVOn4sqVK7h06ZLrPwAvo9frAQBhYWEAOH+8SWtrK5KTk9HQ0IDY2FjOGy/x/PPPY8aMGZg8ebLZ/Zw/7sOg40JXr15Fa2srIiMjze6PjIyEVqv1UKvkpeNztPUZa7VaBAQEIDQ01GZNREREl+ePiIgwq7F8ndDQUAQEBNis6fhf7vNcCIGlS5fiwQcfRHR0NADOH2+Qn5+PW265BUqlEgsWLMCuXbswcuRIzhsvkJycjJMnTyIpKanLNM4f97mpr17uLgqFwux/IUSX+6h3nPmMLWus1buiRnw/WE/u83zRokU4c+YMMjIyukzj/PGc4cOHIy8vD7W1tdixYwfmzp2L9PR0aTrnjWeUlpbihRdeQGpqKgYMGNBtHeeP67FHx4XCw8Ph5+fXJe1WVVV1ScbkHLVaDaDrL4rOn7FarYbJZIJOp7NZU1lZ2eX5q6urzWosX0en06G5udlmTVVVFYCuv8zkZPHixfjss89w8OBBDB48WLqf88fzAgICcPfdd+O+++5DUlISRo8ejTfeeIPzxsNyc3NRVVWFmJgY9O/fH/3790d6ejr++te/on///t32lnD+9B6DjgsFBAQgJiYGaWlpZvenpaVh3LhxHmqVvAwdOhRqtdrsMzaZTEhPT5c+45iYGPj7+5vVVFRUoKCgQKqJjY2FXq9HTk6OVHPs2DHo9XqzmoKCAlRUVEg1qampUCqViImJkWoOHz5sdlhmamoqNBoN7rrrLtd/AB4mhMCiRYuwc+dOHDhwAEOHDjWbzvnjfYQQMBqNnDceNmnSJOTn5yMvL0+63XfffXj66aeRl5eHH/zgB5w/7tJ3455vDh2Hl7/33nuisLBQJCYmiqCgIHHp0iVPN81n1NXViVOnTolTp04JAGL9+vXi1KlT0iH6q1evFiqVSuzcuVPk5+eLp556yuohmIMHDxb79+8XJ0+eFA8//LDVQzDvvfdekZWVJbKyssSoUaOsHoI5adIkcfLkSbF//34xePBgs0Mwa2trRWRkpHjqqadEfn6+2LlzpwgJCfHJQzDt8dxzzwmVSiUOHTokKioqpNv169elGs4fz1mxYoU4fPiwKC4uFmfOnBEvv/yy6Nevn0hNTRVCcN54m85HXQnB+eMuDDpu8Pe//10MGTJEBAQEiP/4j/+QDr0l+xw8eFAA6HKbO3euEKL9MMxXX31VqNVqoVQqxU9/+lORn59v9hyNjY1i0aJFIiwsTAQGBoqZM2eKkpISs5pr166Jp59+WgQHB4vg4GDx9NNPC51OZ1Zz+fJlMWPGDBEYGCjCwsLEokWLzA63FEKIM2fOiIceekgolUqhVqvFypUrfe7wS3tZmy8AxJYtW6Qazh/P+fnPfy6te26//XYxadIkKeQIwXnjbSyDDuePeyiE8MXTHBIRERH1jGN0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhIthh0iIiISLYYdIiIiEi2GHSIiIhItv4/AXOdJXK2N8UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df['review_len'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Houston, we have a problem, let's try to fix our outliers..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "295854    Battlegrounds&gt &gt &gt &gt &gt &gt &gt &gt &...\n",
       "354570    Solo&gt All Map&gt Play&gt Wating other player...\n",
       "Name: review, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['review_len'] > 10000]['review']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.review = df.review.str.replace('&gt', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: >"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjEAAAGdCAYAAADjWSL8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQMElEQVR4nO3de1gU590//jcK4qGwj2hgQ0MT2/Kzppg0xRSxabVV0VRCU/vUJCR80yfWaE00NPrYGnugaQPGNGojiTHGRiNa2j4JadIkK6AGQziIKAoe8IQKygLqspx3Ody/PwwTl8OyA7s7M7vv13XtdenuZ2c+w+zOfuae+77HRwghQERERKQxw5ROgIiIiGgwWMQQERGRJrGIISIiIk1iEUNERESaxCKGiIiINIlFDBEREWkSixgiIiLSJBYxREREpEm+SifgKl1dXbhy5QoCAgLg4+OjdDpERETkACEEGhsbERoaimHD7Le1eGwRc+XKFYSFhSmdBhEREQ1CZWUlbrvtNrsxHlvEBAQEALjxRwgMDFQ4GyIiInJEQ0MDwsLCpN9xezy2iOm+hBQYGMgihoiISGMc6QrCjr1ERESkSSxiiIiISJNYxBAREZEmsYghIiIiTWIRQ0RERJrEIoaIiIg0iUUMERERaRKLGCIiItIkFjFERESkSbKKmI6ODvz2t7/FhAkTMGrUKHz1q1/F888/j66uLilGCIGkpCSEhoZi1KhRmDFjBo4fP26zHIvFgmXLlmH8+PEYM2YM4uLiUFVVZRNjMpmQkJAAnU4HnU6HhIQE1NfXD35LiYiIyKPIKmJefPFFvP7660hNTcXJkyexbt06vPTSS9i0aZMUs27dOqxfvx6pqakoKiqCXq/H7Nmz0djYKMUkJiYiIyMD6enpyM3NRVNTE2JjY9HZ2SnFxMfHo6SkBAaDAQaDASUlJUhISHDCJhMREZFHEDLMmzdPPPHEEzbPzZ8/Xzz22GNCCCG6urqEXq8Xa9eulV5va2sTOp1OvP7660IIIerr64Wfn59IT0+XYi5fviyGDRsmDAaDEEKIEydOCACioKBAisnPzxcAxKlTpxzK1Ww2CwDCbDbL2UQiIiJSkJzfb1ktMffddx/27t2L06dPAwCOHj2K3Nxc/OhHPwIAVFRUwGg0IiYmRnqPv78/pk+fjry8PABAcXEx2tvbbWJCQ0MREREhxeTn50On0yEqKkqKmTp1KnQ6nRTTk8ViQUNDg83D3SwdnXjz0/M4U9NoN66tvRNbD5zH2domN2XWv64ugbc+q0BplVl6rvJ6C37zzjHsLrzU7/v+r7gKuWeu2l12ubERl6612I354OgV7D1ZIy9pFzlWVY/tn1Wgq0sosv7jV8zYlluBjs6ugYOduKzC89eQd9b+vuyLtaMLaQUXceFq84Cx5+qasPXAebS1dw4Y66iL15qxPrMc2z+rQH2LFU2WDrydfwG1DW3o6OzCm5+ex4krDdiRdwEllfV2lyWEwK7Cizh04brN8jftPYP6FisAoMrUgtdzzqGhrd3hHOsaLdiRd0HWe4bqapMFG7JO41xdE17POQdzyxfrvtZkwes551Db0IZrTRb8Ysch/C23wqHlVl5vQdaJGgjh+u9Hi7UDL+05hdXvHkO1uVV6vq29EzsLLqLyeguKL5qwM/+ClE9plRnvFFf1t0gAQLW5FW/nX0CLtcOhPA5WXMdPXvsMVSb7x7FuBeev4cSVvn97Ll5rxpacc2i2OLbubpXXW7Al5xwanfwZuvmzoGWy7mL961//GmazGd/4xjcwfPhwdHZ24oUXXsAjjzwCADAajQCAkJAQm/eFhITg4sWLUsyIESMwduzYXjHd7zcajQgODu61/uDgYCmmp5SUFPzxj3+UszlO9/on57Eh+zT+/OFJXFg7r9+41H1nkbr/LF74yH6cO2QcuYw/fnACAKRcvrduPwAgvagSE/UBiLzddl+dMjZg5b+O2rynJ1OzFXM2HrAbU9vQhmV/P2I3xp3iUj8DAIwdMwI//taXZb+/rb0TJZX1mHL7WPgO7//8oLNL4PgVM+68NdAmbt4ruQAAf99heGzq7bLXfzNHl2Xt6MJDbxQAAEqTYhAw0s/hdbyZex7rDOUABt5/M1/OAQBca7biN/d/w+F12DN7wwFYO24UaR+WVuO2saORceQy/pZbgZ9PuwN//vCkTby9HD87ew1rMsqkuPbOLkx/6RMAwJHKevzt5/fiwVc/w9UmK05cacArj9zjUI6PvVmI8ppGFFZcw2uPRg5iK+Wb8udsAMBf954BABy5ZMKWhCkAgKd2H0bB+ev4d8kVBIz0xcGK68g+WYNpXx+Hb+gD7S63+7jw5v+bgll3fnGMzzxuxB8/OIFXHvkWIm8Pcso2vLSnHG99dgEA8OmZq8j99Q8BAK/uP4tN+87Cd5gPOj4/2Qj9r1GYOSkED6Te+MzrdSPx3a+P73O5P079DLWNFpysbkTK/MkD5rFgSz4A4L4X9w/4Gb9S34qHP/8u9RU7e/0BWDu7UGlqwZ8fHHjd3R5IzUV9SzvKaxqxfsG3HH7fQLo/C++XXMFHz3zPact1N1ktMf/4xz+QlpaG3bt34/Dhw9ixYwf+8pe/YMeOHTZxPW+fLYQY8JbaPWP6ire3nNWrV8NsNkuPyspKRzfLaUoqTQ7FHb7kWFxfWq2dyDldB0uHc85oTxntt1j1dQZSbR64cr9c3zpgjLnVfWencpQb7bek9eepXYfx8BsFWJ912m7cOsMpxKV+ht++V9bn6yeqndeKeHKAZVlvaqlpbJN3hniw4vrAQT0M5bPfU3cBAwBFF0zI/rxF78K1FpT1czbcn4qrtq2ixps+4/tO1QIArjbdaJHJO+d4q1X5562yWSeUa2389PMW0/bOLhScv7HPTlY32Oy/qusDf1+7Hbpouw+f3FmMy/Wt+H/bDjoh2xtscjN9kVv+uWsAIBUwAHC+zrYl0F5LeG2jBQBw4HSdU/K82UDHvO7vWlGFvO9A/ectaQWfb7uzdH8WnHm8UYKsIuZ///d/8Zvf/AYPP/wwJk+ejISEBPzqV79CSkoKAECv1wNAr9aS2tpaqXVGr9fDarXCZDLZjamp6f2lr6ur69XK083f3x+BgYE2D7X58Fg1/nPsyoBxrdZOmybgmy1PP4LH/3YQf/rPCWenR0O09/Mfu+15F+zGbTlwHsCNli6ibo1t7eh04aXM9IP9Xx52BkvH0C+DEsklq4hpaWnBsGG2bxk+fLg0xHrChAnQ6/XIysqSXrdarcjJycG0adMAAJGRkfDz87OJqa6uRllZmRQTHR0Ns9mMgwe/qOwLCwthNpulGC0xt7Tjx69+hqd2H8bTu4+gyc410UvXWjDp9wbc/XymTVyrtRPtnV3SGV1awY0DkjP6T5B815ut2HeqxqU/Ou7QJLP1hVzjSn0rJidlYv5rn7lsHUaN930g6ousIuaBBx7ACy+8gA8//BAXLlxARkYG1q9fj5/85CcAblwCSkxMRHJyMjIyMlBWVoaf//znGD16NOLj4wEAOp0OCxcuxIoVK7B3714cOXIEjz32GCZPnoxZs2YBACZNmoS5c+di0aJFKCgoQEFBARYtWoTY2FhMnDjRyX8C19u07wyO3tSx0NLef+Exc/0n0r+7O/62WDsw6fcG3PfiPpvYFf88iq+v+bjX5Y/Dl0y42mQZeuJ96OoSNs273a7Ut6LyumOd35Sys+AiYjbkSJcKjlwyYftnFYPqqBj7yqd4YvshvJ1/YcDYZkuHYp2F7Wlr78TUlL1OX257Zxcee7MQ6zPLnb5sZzhf14Tii867rOWI9k77+/+j0moAwNGbOtjbY+3owt6TNQ539tR6sU3UH1lFzKZNm/Df//3fWLp0KSZNmoSVK1di8eLF+NOf/iTFrFq1ComJiVi6dCmmTJmCy5cvIzMzEwEBAVLMhg0b8OCDD2LBggX47ne/i9GjR+ODDz7A8OHDpZhdu3Zh8uTJiImJQUxMDO666y7s3LnTCZvsfnL6Gtx8sHvw1c9wpb5V6u1e02BbmLxz+EZP/DkbD0ijPgrOX8P81/Kkzn2njA3YlluBdgdabB57s9Cmn0FfnvlHCX7Xoy9HZ5fAtLX78L11+9Fqdd7oE2f73XtlOF3ThJSPb3T4/MlreUj64AT2HO/dWby2oQ2GMmO/B/8rnxdChrK+O5rfvJxv/mEPfvp636Pq+rK78JJbRoDYG3FRUlmPnEH2G9hz3Ijcs1fxyr6zTh2B197ZZbcYdPR79sOXc/DTzXmKF91HLpmws+Bir33dPRrKnr9klmPhjkN4YnvRgLEt1k7c+0I2Xt1/btC5atX1ZqvD36UzNY34R9El6TOm9lZuIQT+919HkfLRyYGDPZis0UkBAQHYuHEjNm7c2G+Mj48PkpKSkJSU1G/MyJEjsWnTJptJ8noKCgpCWlqanPQ07WqTBV/y7707pq3dhz89GDHg+00tVtyqG9Vr2PPcjZ9K/1543wS7y8g9e1U6I+zPB0d79+m5uZPx1SYLwoJGD5jvYFVeb8GnZ67ip5Ffhr/v8IHf0IeeLWHn6noPEZ61PgcNbR34Y9w38fi0OwDcaIUaNsy2Y3lhxXV8dvZqv6MhDJ8XSEcu1cvKseJqM756y5dkvceZHnz1xmWNvN/8EKH/NUrWe2/++85an4PSpBg70Y6xdnRh2tq9uCVgJD52YCTF/w0w1BYA3j18Gf9z3x0IHOmHE9VftGa6q8P5T167UdiGBPjbPP+t57Owf+UMTBg/pt/3/uvQjf5URRcca1G63jxwYaRlfXXGN5QZsSStGI9N/YpDo4Fmb7gxmnL4sGH478jb8LfPHBt67iztnV0QAhjh61jbwvmrzfjX55/z1T+a5MrUVI33TlKBanMrpvw5WxrC2FPPlo/BOH7Z8WZqNZu5PgfPZZS69KzydE0jGj4/q99ffqOzbpWpBZF/zsJf9vS+RPLom4X9LuvmOSOW7iq22x/qZl0Onj0KIZD0/nG7c/oMhTP6UXSP6hmKU8YGXG2yDjjaSo4N2adx/+dF/t9v6vQalZw96GXWNrThzU/P99sxvy99FdHvHbk86Bxc7fWcc6q7PPXLXcW9nltnOAXgi/6DjjpWVQ8AyD5RO+S8HNXVJRCdsheRf8pyqNUcgMNxzlZ22YyXM8sdnmvH1VjEuMhf9pSjrtGxfindrSeOxnuz7iJrMJOzOSr7ZO+D14asMzC1tCN1/1mHl3O92WozAumjUiNe/8S5xVf++WvYnncBz2WUOnW5atLe2YVf/aPEJcvua1hsm50+a4D9oixh20H8+cOTePafJUNNTdX6ugSrJK13UG9t78TVJisaLR2o6efEoUv0/xuR7cYh/LGbcrFp31n8NfuM29ZpD4uYQTKUGbHs70dsZl+su6kzber+s1j298NKpEZOlN/H3Az/72+OzYfRV6vBtT6a9Wsb2vDPQ72HW5+va8L+U/bPBhtab/r8eWgR/O7hqj5bK9Soe16YvQPsN61zddFwXOY8P850eoAZ15VibGjDvS9ko+B872PSL94+5PZ8/uXAJVt3YBEzSEvSivHB0SvYfNOZdX2PJuTCQUwGNljd6765iXFX4UW3rd9T9TX3hSMTZbV3dtm9zHSzn7yWh1X/d6zX8z98OQf/s73IZip8ex76fHZRT9PzewXA4ctyzlLhwK0VyDN0TwKnVts/n8lYaWrpZ8UiZojsnf0KAbfdM+Unr32G0iqzNJEaAGkadXK/BhmdQwea6bPUwf5M573kh/bNT88j4g973LrOI06caZjUp68+Ps1D6PNhbmnH4Usmt4wy9HYsYlzsMQfPxoeqrb0Law3ePdSOvEPPeyIRDVVfUyUM5ZLWrA05mP9aXp/968i5WMS42DEHJ69SCk8USE2aLR34d8lllwxzVuOEg6QOzr5DdHcLvdo6QHsiFjFETuYthaErtnP1u6V4Jr0Ev0zrPWR2qIp5SYjIqZxd/A0GixgiD7bLRfPHuMr7n0+mmOfkO/YCQLvK50CiwXN0XiVyrslJmdi0V9mh1ixiyC4hAJNKeqGTd3l6N6coIMf0nLunxdoBkwO3b3Amn4FD3MLdE5a+nHXarevriUUM2bXyX0dxz5+y8Oan5wcOJnKi/xyzfwsMV9LSif3zH5xQOgUAgJr+ZBF/2AOTjFmTPcWxqnr8f7/9GC/tOaV0Km7DIkYBPmop2THwgafj886QHBFCZMvc2q6KIbTuvsePFriqD3f3jXbV6s//uXGcdtVtWW6+T55asIhxA6GqcxRSlud8FjxnS+Q7dOE67v5jJp7951GlU6EefFx0lvhcRim+8TsDjl9R94hTVxpoBnElsIghcqPDF+uVToGcoPseWhkqvlEjOVf3TVZflXH/NE+wPus04lJz0WLtcFkL11D4Kp0Aue7M4WYq/Ox5JWdPl9/R1bsTn1r6SJB28PhA/Xnl89FH/1dchfFf8lc4m97YEuMlVNQNh5zo6d1Hej3n6j4SZ2ubXDIZHRGpV3unOktdtsQ4kQr6+JGHSN2nzibrE1ca8KNXPsVIv2H4zoRxSqdDpDlqGtjhCdgSMwjFFznzJ9k31Hr2vZIrTsnDHkeK7p7H25zP7+Dd1s6J45yFP2ra8lGpEbUNbb2efzlz8POl3PwZ4MmwPCxiZDp04Tp+ujlP6TSIyEnezr+gdAo0BEoUgWsNvedhyT/v/FmmaWAsYmTK7zEdOodPU0/8TKhUPz92v//3cffmQZrXalXffCneikXMEP3zUBU+O3tV6TQGjU2Xzse/KXk8XgIjlWAR4wSPvlkoK94dQ6oHTcWpKaXyeovSKQyJEMIlZ45qmK1WDh9+uJ1G639JrX12qX8sYjwIv5eu8fjfDiq6/qHu12f/eRSTfm/AmZpG2e9VdcFNTqOVveyujyOLHO1gEeNB3H330m6efoZ7/mozrJ3aHY3TPavstlz33GOHh3/SuqW7eAf1d4qrcLSy3uY5NdZ2nCdGAa76yT9kb+i3m+sMNX7Yh+KwjGH1lo4uNLU5d2ZeT+HZ5S65Sn2LFfFbC3GiusEly+95uPq4zOiS9biTqdmKa80WfD04QPZ7889dw4p/aeO+YGyJIa9ztKre5euY8ZdPnLYsD6sHPYKzi3RetrNvc865fgsYi0It0Gp3z5+yMGv9AZytlX8Z+Wxdkwsycg0WMeR1qs29J6oiupmcYfK1je7/PJla1HfbB1f2I7HYmVxx074zLluvo2a8tF/pFPpVWHFd6RRcikWMG3japRXSppIe17fJOb7zwl6lU/BqSs8efb3ZigvXtD2CUcvYJ0YFlGhINpQZcdvYUZyYzYucMspvViYi+9o13OlfDrVe8GQR44XKLpuxJK0YAPA/371D2WTIqxRWXEdXl8CwYcocEi0dg5svh11WtEPt+4ot887Fy0le6JwbOm2xhYf6887hKkXWe73Zijt/v0eRdQ9G1okaGMqqlU6DSNVkFTF33HEHfHx8ej2eeuopADc6diUlJSE0NBSjRo3CjBkzcPy47X1JLBYLli1bhvHjx2PMmDGIi4tDVZXtQc1kMiEhIQE6nQ46nQ4JCQmor68f2pYS9aPnXAjkWqcHMemeM2QcuYzOLm0U19bOLix6+xCWpB2GWYWdeInUQlYRU1RUhOrqaumRlZUFAPjZz34GAFi3bh3Wr1+P1NRUFBUVQa/XY/bs2Whs/OKglZiYiIyMDKSnpyM3NxdNTU2IjY1FZ+cXzbzx8fEoKSmBwWCAwWBASUkJEhISnLG9RL1knqhBtblV6TRIxdx9CaD9pmHDTVbOOUTUH1l9Ym655Rab/69duxZf+9rXMH36dAghsHHjRqxZswbz588HAOzYsQMhISHYvXs3Fi9eDLPZjG3btmHnzp2YNWsWACAtLQ1hYWHIzs7GnDlzcPLkSRgMBhQUFCAqKgoAsHXrVkRHR6O8vBwTJ050xnbT5zzh+myrtRNXzK342i1fGvQyolP2OTEjz6Hy7gVEHsFZs55742X8QfeJsVqtSEtLwxNPPAEfHx9UVFTAaDQiJiZGivH398f06dORl5cHACguLkZ7e7tNTGhoKCIiIqSY/Px86HQ6qYABgKlTp0Kn00kxfbFYLGhoaLB5qIX3fazcK2ZjDma+nIOiC549H4Kz1Da24aldh5F//prSqRCRhqixSBp0EfPee++hvr4eP//5zwEARuONaZpDQkJs4kJCQqTXjEYjRowYgbFjx9qNCQ4O7rW+4OBgKaYvKSkpUh8anU6HsLCwwW4aaUzl9RuXgj4qZSdIR/zuvTJ8WFqN3//7+MDBdmitFU+to1bUkJbGdiUpRI3f+UEXMdu2bcP999+P0NBQm+d7Tp8thBhwSu2eMX3FD7Sc1atXw2w2S4/KykpHNkMRaj2YAuo4oKqRErOyukqVSTv9fzzp705EzjeoIubixYvIzs7GL37xC+k5vV4PAL1aS2pra6XWGb1eD6vVCpPJZDempqam1zrr6up6tfLczN/fH4GBgTYPV7BXgFyu186PA8nT3qnCU5BBUuPZVH+iktU5G66G/oREHm1QRcxbb72F4OBgzJs3T3puwoQJ0Ov10ogl4Ea/mZycHEybNg0AEBkZCT8/P5uY6upqlJWVSTHR0dEwm804ePCgFFNYWAiz2SzFaJ2aW2LItdRQ6DZraLSLlgoub8IbVpJayJ6xt6urC2+99RYef/xx+Pp+8XYfHx8kJiYiOTkZ4eHhCA8PR3JyMkaPHo34+HgAgE6nw8KFC7FixQqMGzcOQUFBWLlyJSZPniyNVpo0aRLmzp2LRYsWYcuWLQCAJ598ErGxsRyZRESa0VdH8/oWK8b4+8JvOOcZ1bKh1HCs/5xLdhGTnZ2NS5cu4Yknnuj12qpVq9Da2oqlS5fCZDIhKioKmZmZCAgIkGI2bNgAX19fLFiwAK2trZg5cya2b9+O4cOHSzG7du3C8uXLpVFMcXFxSE1NHcz2EZEMjvRhI8f87PV8m/9fqW/FtLX78LVbxmDvihnKJOUk/ISQWsguYmJiYvq95bqPjw+SkpKQlJTU7/tHjhyJTZs2YdOmTf3GBAUFIS0tTW5qmqWm3wz+gKlPf983ZzO1WLFgSz7m3XWrW9bnbbJO3Ojrd66uWeFMXINX/nrzpMuhav1pYJumG3jSB5k81468C6gytWJLznmlUyEiNzpyyeS2kyVnYxFDpBFHK+vxwocn0NjmmnvpdHYNHOMMWmjt0+jxnGhQfvJaHgxl/c/DpmayLycRkTJ+/OpnAIC29i786cEIt6xTA/UGeQFXfw5ZtAL/OVaN+ydr71IyW2IU4Kz7ZKgRDwauV+7Gu0DHby1ETQMnnCO62c3H8FZrp51Iz6LGwzuLGBoynq0Dmz85p3QKLtFk6cDzH5xQOg2PpsYfBnLch7zdiaJYxHiJ/lp/nNGZ63qzdcjL0LoXDaeUTsFlTC3cv6Qdajqn4gme67GIcYO2DtvmRjXeCXSwahvaMG3tPqXTIBl4yY+8nQDwyBsFWLi9SOlUALDYGQp27HWDD48p39zoqi9J/vlrrlkwuYyziuj+lqPVoZpyeMM2erJqcxuqzTf6enlTnxZPxJYYIhoU/o57DjYEqMdQvlfO+k5q6fPAIobIy7D4IFKv3LNXsf2zCqXT0AxeTlKAJw+xJu3ypL5aWuJNR4P2zi7e/HIAz2WUAgC+cWsgpn51nMLZqB8/TUREClNDIVPb0IbFOw8h98xVlyz/7fwLCF/zMXJO17lk+Z7msqlV6RQ0gUWMTM6YMr213bYjmTt6ph+5ZHL9Sm7CSxbkSbxh9MhzGaXYc7wGj20rdMnyf//v4wCA5X8/4pLlk/P0dfhW61eARYwCztY2uX2dnnbnXNZI2uUDoMXagb9mn8EpY4PS6dDnLtdzZmat88bjIosYInK7lzNPY0P2aczd+KnSqThN/QCTAqp5WLbczLyhZcrbaLVPHIsYL6SFuwhT/w5WXMdjbxbCaNbumfOxqnqHY/cc18bddT91UV8SrVBxjUYOcGT/qbEQZxFDpEG5Z6/it++VDeq9KjwO2XX4Ur3SKZAHc3cLRFs7J9dzJhYxRBp1vdnilvVorehxBi/cZHKTC9dalE7Bo7CI8XJqOVg7o5nSU+ffcXYRocYmYdKWoVyRFkKgq0vjn0HPPNRoEosYUgy75hANzNOKzoa2DvzolU9lFTKeeoJCQ8cihmzwUEHUm4fVEb1cqXfvxGqnjI2obtBux3RSDxYxRF7kdE0jrvQ3qknmD3WlSVvX9lmg9+8P7x9XOgUAN4apu7vjqxDaHV5MvHeSV7r5YO7pZ5gAcLm+FQtez1c6jSFJ3Xem13ODGbUTs+GAE7K5ofJ632fvav1IqTUvuqG+xYpvPZ+FMSOGK50KaQiLGPJ4f9lTjstubi531MnqgWesrW+14tX951yeiyP3tDlYcc3leXgqOZdPvLG/WEllPQCg2TrElhhv/ON5MV5OIo/X2UcHQkeKB3cou2weMKa90z1tCNea7c84CwD7y51w8z4v/Y3ZknNe6RTs8rQOxOQdWMSoAHveu19ieonSKRARaYZaZ3pnEUMuofZzuvbOLqVTcJinnSGzaFe/3DNX+2zB1ASZ3xe1buWKfx1VTYtxNzUeiljEkFOxlz+RPGo8wX1sWyF25l9QOo1B+d2/1THSyhkefbPQbetSY4HiCBYxROR2Wj1gepMPjlUrnYImuLIIve5APzVn0eoJKIsYIrJLqwe3nlg4kbOosPHMa3GINXkMNTbLu9vekzVKp0CD0NE1tD5a5tZ2/GJHEeK+9WUnZUReTUMVv+yWmMuXL+Oxxx7DuHHjMHr0aHzrW99CcXGx9LoQAklJSQgNDcWoUaMwY8YMHD9ue43SYrFg2bJlGD9+PMaMGYO4uDhUVVXZxJhMJiQkJECn00Gn0yEhIQH19fWD20o30Grny55n2SwE1KffGXb7sHDHIRdmQt2c+XV/p/gykj86NaRlbMk5h6ILJvzuvTInZUXO0tjWjsOXTEqnIYuW+hXJKmJMJhO++93vws/PDx9//DFOnDiBl19+Gf/1X/8lxaxbtw7r169HamoqioqKoNfrMXv2bDQ2NkoxiYmJyMjIQHp6OnJzc9HU1ITY2Fh0dn4xyVF8fDxKSkpgMBhgMBhQUlKChISEoW+xi1RcbVY6BVXRalGnRh9qrW+CSne9qwv0wS6/vKZx4KABtAx1gjiZOMLMcT965VPMfy1P6TQ8lqzLSS+++CLCwsLw1ltvSc/dcccd0r+FENi4cSPWrFmD+fPnAwB27NiBkJAQ7N69G4sXL4bZbMa2bduwc+dOzJo1CwCQlpaGsLAwZGdnY86cOTh58iQMBgMKCgoQFRUFANi6dSuio6NRXl6OiRMnDnW7nU6lx21FrPjnUbxzuGrgQKLPWTu6sHina1uR+LMrn6WjE69/ch4zJt6idCqa1d/tOcg5ZLXEvP/++5gyZQp+9rOfITg4GPfccw+2bt0qvV5RUQGj0YiYmBjpOX9/f0yfPh15eTcq0eLiYrS3t9vEhIaGIiIiQorJz8+HTqeTChgAmDp1KnQ6nRTTk8ViQUNDg82DlDGYAsbS0YW8c1dh6XDvGSWpw4elV5wzG7CTDNSS6CmdnQeyLbcCG7JP48evfubydXnHX5ScTVYRc/78eWzevBnh4eHYs2cPlixZguXLl+Ptt98GABiNRgBASEiIzftCQkKk14xGI0aMGIGxY8fajQkODu61/uDgYCmmp5SUFKn/jE6nQ1hYmJxNI4WtyShD/NZC/EFD12LJedx9OcTZPiqthrVDOxMoOuq0ceiXuryZu6+qu/Iy/it7z2Bj9mmXLX+wZBUxXV1d+Pa3v43k5GTcc889WLx4MRYtWoTNmzfbxPWcnlgIMeCUxT1j+oq3t5zVq1fDbDZLj8rKSkc3i1Sgu/UmvYj7zdN5YufxpbsOY+3HQ+ucqzaanbFXJvbfc8y1ZisuXGtROo1eZBUxt956K+68806b5yZNmoRLly4BAPR6PQD0ai2pra2VWmf0ej2sVitMJpPdmJqa3kNF6+rqerXydPP390dgYKDNg4jIGRwpvGobLa5PxI267yrdF3cXoq5cH0sYbZNVxHz3u99FeXm5zXOnT5/G7bffDgCYMGEC9Ho9srKypNetVitycnIwbdo0AEBkZCT8/PxsYqqrq1FWVibFREdHw2w24+DBg1JMYWEhzGazFONJPPHMlEht+GPlvawdXcg+od45lPgbMHiyRif96le/wrRp05CcnIwFCxbg4MGDeOONN/DGG28AuHEJKDExEcnJyQgPD0d4eDiSk5MxevRoxMfHAwB0Oh0WLlyIFStWYNy4cQgKCsLKlSsxefJkabTSpEmTMHfuXCxatAhbtmwBADz55JOIjY1VfGSSJ3zYbt4GLbekNlk68CX/gT/CF65x+LuaXLjajLN1TW5fr5Y/6zQ0r+w9g9T9Z522PLXe0dkbySpi7r33XmRkZGD16tV4/vnnMWHCBGzcuBGPPvqoFLNq1Sq0trZi6dKlMJlMiIqKQmZmJgICAqSYDRs2wNfXFwsWLEBraytmzpyJ7du3Y/jw4VLMrl27sHz5cmkUU1xcHFJTU4e6veQhii+aEPGHPUhbGDVgbDk7J6rKx2V9d86nL7Dgcq6MI5eVTkH1tPqZk33bgdjYWMTGxvb7uo+PD5KSkpCUlNRvzMiRI7Fp0yZs2rSp35igoCCkpaXJTY+GSGsnGC9llg8cRETqp+CPqFZ/wIk3gCQPorH6SzPccYD3lpEw5Hk8ZfZirX4DWcR4IU/50nkCrR44+tPRKX+ulCOXTPi9G+YHcmYrI8/c3Yt/b+oP72JNHsfc0g4A0I32UzgT73K5vhUzX/5EdpHsjgKGyNPVNLR53EmRI1jEeLmhfujV1ku/vbMLdz+fCQA488L98Buu8sZGDzrqbMk5h7Z2z5u1ljyfubXdJct15+ExKnmv+1amIio/whPJ03DTwajBRQcmb6X1fivqKrepJ6XuRyUEZ+3VMrbEyNRfU3mrxu/9QmRPZ5fA9Jf2K50GeaDii9fxJX/3Xvr18WE/G0/Blhgn+fvBS0qnQOQyl02tqDK1Kp0GuVlru2tPzmoa2vDTzfmYs/GAS9dDnotFjJO4+stOREMzlBNvJc/a3bXuvtaz57hrp+qvMqnvhoKkLSxiiIh64JUGIm1gEUNOxevMRKQpQzxmqWyAptdhx14iIpkuXuNlEC3zgXpa22ob2/DRsWql09AsFjFE5BTe1Ap3uZ6dnMk5Hn6jAOfrmpVOQ7PfX15OIiJFuas5nq3+5CzO/CypoYDRMhYx5DH2l9cqnYJH0ugJWi+esh3eaKiFrr33KzXJHjkHixgv5Kkd0c7VNaNeY7P0Wgdxw0TyfGr/jqrtdiNyaT1/19BmMccixstp9Tpof1b886jSKcjy2/fKlE7Ba1zmZH2q42nHH3I/FjFez7OOIiWV9UqnQCp1orpB6RSoh5uPPloqaNo7u9Bi7VA6DQKLGCJysc2fnFU6BSKn+rjMiDt/vwdNFmUKmdrGNjQrtG614RBrInKpfx6qUjoFchMhxKD7m2jx1g7HL5udm4gDrjVb8J0X9joc399Niz0FixgiUpRnH2K1wxmjdKpMrQgLGu2EbLRjsJ/fT8/U4T9H5U9yd7TS/YWTmrGIcRYNXc+1x9OrdhocR37gPHmoKgezkLMlbDuodAoegX1iaEh4bKch88IK4cJVbU9wVnld27ddEFrqRUx2sYiRyRXHW85ZQORd1u05pXQKQ2Jp71Q6BafifE3axSKGiIgUoYYGESGATfs4gk6rWMQ4CxtTiAbFaOYkdOReN7d+q6COUoXsk7VYn3Va6TRkYxFDRIqqabAonUIvamghINfhOWffXtl7BqZmq9JpyMIixkmqPGRKc3bPoV74g04uwg626tPepa3+QSxinKSdHcO0hcUa0eBp7GxHY+mSDCxivNA7xZxBlZyPJ9U0FPz40GCwiFHYd9fuw6EL1926zr2nat26PiIib8TC3vVYxCjscn0r3s6/qNj6veFL5gWbSETklWQVMUlJSfDx8bF56PV66XUhBJKSkhAaGopRo0ZhxowZOH78uM0yLBYLli1bhvHjx2PMmDGIi4tDVZXt5Q2TyYSEhATodDrodDokJCSgvr5+8FtJRINWdPE6rB3s86VmGUcuD3kZaj6hYZeWL3jy7T0GQ3ZLzDe/+U1UV1dLj9LSUum1devWYf369UhNTUVRURH0ej1mz56NxsZGKSYxMREZGRlIT09Hbm4umpqaEBsbi87OL2aAjI+PR0lJCQwGAwwGA0pKSpCQkDDETSV34NfL8wgBrPjXUaXTIDvMre2KrHeoo4t4vKChkn0DSF9fX5vWl25CCGzcuBFr1qzB/PnzAQA7duxASEgIdu/ejcWLF8NsNmPbtm3YuXMnZs2aBQBIS0tDWFgYsrOzMWfOHJw8eRIGgwEFBQWIiooCAGzduhXR0dEoLy/HxIkTh7K91INaz76EELB2dsHfd7jSqRCAY1W8c64zDXSj1crr6p+y4ZW9Z7Aj74LSaQzZUAoxlR4+vYrslpgzZ84gNDQUEyZMwMMPP4zz588DACoqKmA0GhETEyPF+vv7Y/r06cjLywMAFBcXo7293SYmNDQUERERUkx+fj50Op1UwADA1KlTodPppJi+WCwWNDQ02DzcSa3FgFY9tKUA3/z9HqecYbIpmrTGcNyodAp2+QBYn3Ua1zQ2MVo3Drn2HLKKmKioKLz99tvYs2cPtm7dCqPRiGnTpuHatWswGm986UJCQmzeExISIr1mNBoxYsQIjB071m5McHBwr3UHBwdLMX1JSUmR+tDodDqEhYXJ2bQhYw3jXAcvXEdHl0DO6Tq7cTwWEdFAmto6lE6BXERWEXP//ffjpz/9KSZPnoxZs2bhww8/BHDjslG3nndkFkIMeJfmnjF9xQ+0nNWrV8NsNkuPyspKh7aJiGggbGl1DXf9XU0tti267Z3K7lCl+jB5oiENsR4zZgwmT56MM2fOSP1keraW1NbWSq0zer0eVqsVJpPJbkxNTU2vddXV1fVq5bmZv78/AgMDbR5E5D78naeh8KZbEExN3qt0Ch5jSEWMxWLByZMnceutt2LChAnQ6/XIysqSXrdarcjJycG0adMAAJGRkfDz87OJqa6uRllZmRQTHR0Ns9mMgwcPSjGFhYUwm81SDBGRO+09yQkiPdVQS6fB9K9pbe8cOIgcImt00sqVK/HAAw/gK1/5Cmpra/HnP/8ZDQ0NePzxx+Hj44PExEQkJycjPDwc4eHhSE5OxujRoxEfHw8A0Ol0WLhwIVasWIFx48YhKCgIK1eulC5PAcCkSZMwd+5cLFq0CFu2bAEAPPnkk4iNjeXIJCewdPDLQyTXh6XVSqegCUNpTPGedhh1yzrR+0qImskqYqqqqvDII4/g6tWruOWWWzB16lQUFBTg9ttvBwCsWrUKra2tWLp0KUwmE6KiopCZmYmAgABpGRs2bICvry8WLFiA1tZWzJw5E9u3b8fw4V8Mpd21axeWL18ujWKKi4tDamqqM7Z3yLTckXTzJ+fwouGUU5fJXv5ENFhan7hNCOCyyb3D4V3dn+efh7R1bz1ZRUx6errd1318fJCUlISkpKR+Y0aOHIlNmzZh06ZN/cYEBQUhLS1NTmqKu2xqUTqFATm7gCHyXNr+cR2Mji7Ht7m20eLCTLRj66fnsc/Ovehyz9ofXUlDx3snOcnVJm3Ol6B23tTZj0hJcr5rl643uzAT7bBXwABATQOLPVdjEePltN6cS6RJvAyrGjxP0jYWMaQgHsmJ1MLdv+VsZXWvsstmrMkoxdUmz2odkn3vJCKivvA3yXsMdO8nUp/YTbkAgF2Fl7D8h19XOBvnYUsMeTz+thI5l9MuQ/PLqYhX9p1VOgWnYRFDREREAICjlfVKpyALixjyeH02fPMMkMiGnMuB/Y26Gcq8UewjQ4PBIoacqqjiutIpEJGLna1tcspybi5b3jl8ud84TqpJ/WER4+WcffKz6p1jTl0eT86IvMPxy2alUyANYhFDRESK4/kKDQaLGJk4AyMREZE6sIiRqcXaoXQKHm9j9mmlUyByKTV28ag2u/dGhmrBWcu1jUUMqc7G7DPSv3mA0RLuKy1z9t2ROzq7cMrY4NRlusKhCyalU6Ah4Iy9REQacfFaM0b6DVc6DYd8fc3HAICf3PNlbHjoW8omY8f+AW7iSOrGlhgvp6Vz5zKOXiAvVt9ixfSXPkFU8l6lU5El40j/Q6dvHn3IkYg0GCxiZOJ8Bco5cKZO6RTIS6jxB/XitRalU3A6Xi6moeLlJFI1Nf6YEJF9be2dA8Z8VFqNc7XNbsiGPBmLGBoS3s2WSD5Pr81f3X/W7v8BIPmjU+5KhzwYLycREZFTdLecnjI22jz/0p5yBbIhb8Aihmz4sNMPERFpBIsYssEShoiUYO/8yZV94zz90p6nYxFDHo8HKfdgJ2zH8WShN35+aDBYxJCq8cBGSrB0dCmdAhE5gEWMTPxRVQ7/9uQu+ziLK5EmsIghVWPdQkpQU8Gsplw8Ue6Zq0qnQEPAIsbL8a7cROonhMBj2wqVTsNhzu7z48pBk8aGNtctnFyORYyX+6jUqHQKRKrT3qmuPjHn6prR2KadE47BFB0sJmgwWMSQx+NIENera7TgdE3jwIE0KILXlIj6xNsOkKZxbj51uPeFbKVTICIvxJYYmXhC5BkaLdppmicior6xiJGJt46Xz1BWrbo+BqZmq9IpkIrxez445682KZ0CeZkhFTEpKSnw8fFBYmKi9JwQAklJSQgNDcWoUaMwY8YMHD9+3OZ9FosFy5Ytw/jx4zFmzBjExcWhqqrKJsZkMiEhIQE6nQ46nQ4JCQmor68fSrqkkCVph7El55zSaRCphqfeo+zJncVKp0BeZtBFTFFREd544w3cddddNs+vW7cO69evR2pqKoqKiqDX6zF79mw0Nn7R6S8xMREZGRlIT09Hbm4umpqaEBsbi87OTikmPj4eJSUlMBgMMBgMKCkpQUJCwmDTJYXtOV4zqPfd3KGRnRvJXdT0UTunodYNK2c6JjcbVBHT1NSERx99FFu3bsXYsWOl54UQ2LhxI9asWYP58+cjIiICO3bsQEtLC3bv3g0AMJvN2LZtG15++WXMmjUL99xzD9LS0lBaWors7BudA0+ePAmDwYA333wT0dHRiI6OxtatW/Gf//wH5eXK3tLdh2Nd3EpFvyVEivift4qUTkE2HifJXQZVxDz11FOYN28eZs2aZfN8RUUFjEYjYmJipOf8/f0xffp05OXlAQCKi4vR3t5uExMaGoqIiAgpJj8/HzqdDlFRUVLM1KlTodPppJieLBYLGhoabB5ERJ5Aa8W8h14tIxWSPcQ6PT0dhw8fRlFR77MDo/HGxGkhISE2z4eEhODixYtSzIgRI2xacLpjut9vNBoRHBzca/nBwcFSTE8pKSn44x//KHdzSCO6ugRarJ0DBxIRkdeQ1RJTWVmJZ555BmlpaRg5cmS/cT07rQkhBuzI1jOmr3h7y1m9ejXMZrP0qKystLu+weKoBVvuOuN6ZGsBXvuEnYPJPVz9uf7g6BXXroDIS8gqYoqLi1FbW4vIyEj4+vrC19cXOTk5eOWVV+Dr6yu1wPRsLamtrZVe0+v1sFqtMJlMdmNqanp3BK2rq+vVytPN398fgYGBNg/yHIUV1wf9XpadJJeaOvYS9wf1T1YRM3PmTJSWlqKkpER6TJkyBY8++ihKSkrw1a9+FXq9HllZWdJ7rFYrcnJyMG3aNABAZGQk/Pz8bGKqq6tRVlYmxURHR8NsNuPgwYNSTGFhIcxmsxRDRETeYdtnFUqnQColq09MQEAAIiIibJ4bM2YMxo0bJz2fmJiI5ORkhIeHIzw8HMnJyRg9ejTi4+MBADqdDgsXLsSKFSswbtw4BAUFYeXKlZg8ebLUUXjSpEmYO3cuFi1ahC1btgAAnnzyScTGxmLixIlD3mjyHI6cofV1ZYAdD4m0gy0x1B+n3ztp1apVaG1txdKlS2EymRAVFYXMzEwEBARIMRs2bICvry8WLFiA1tZWzJw5E9u3b8fw4cOlmF27dmH58uXSKKa4uDikpqY6O11SOx68iDSHJwnkLkMuYj755BOb//v4+CApKQlJSUn9vmfkyJHYtGkTNm3a1G9MUFAQ0tLShpoeyeSqgw8PakRE5Gy8dxIRUQ8cCTc0nOyO3IVFjEy8NmvrV/8ocTi2q0tgyc5i/GWP47MuD2VI+2/fKxv0e4nUhMcdor6xiKEhsci4V8rBC9dhOG5E6v6zLszoCx+X9T0xIhEReQYWMeQ2cgoeIiKigbCIIbfgFXIiInI2FjHkldjxkMiF+PVSlbZ2z73vHIsYmdi/zr3YoZEIMLe2K52CLLUNbUqnQJ/r7OrCnb83KJ2Gy7CIISJSuT9+cFzpFGQpumAaOIjc4mqTFV0efDLIIoaISOVOGRuVToE0yoPrFwAsYshdOGUvERE5GYsYIiKVExrqHNZi7VA6BbqJp58+soghG50qu3iqrmyIaCCv7ectG8h9WMSQjX8UVbps2UqdTbIQInKfKlOL0imQF2ERQzaqTK1Kp+AWnRpqniciGqxdhReVTsGlWMTIxN8+93LV31ttl82IiFzhapNV6RRcikUM2XDlICIfjlAi8ngnqhuUToG8CIsYIiJymtM1TUqnQF6ERQwRERFpEosYcgteSCIaPPbgIuobixhyi57dYbJO1CiTCBEReQwWMaSIRW8fctu62ApEWsfPMFHfWMTIJNiw6zQ5p+sGjBno780BT0RE3otFDCnmT/85MWAM5+UhIqL+sIghhxnKqof0fi3dxI6IiNSPRQw5bEnaYaVTIPJKnGCaqG8sYmTyYRc7IiIiVWARI5Ond+zlFR8iItIKFjFy8Ud+UHygzL2TztY2wtLR5fb1EhGR67GIIY82a/0BpVMgIiIXYRFDREREmsQihlSNV++IiKg/soqYzZs346677kJgYCACAwMRHR2Njz/+WHpdCIGkpCSEhoZi1KhRmDFjBo4fP26zDIvFgmXLlmH8+PEYM2YM4uLiUFVVZRNjMpmQkJAAnU4HnU6HhIQE1NfXD34rSbsG6Gl85FK9e/IgIiLVkVXE3HbbbVi7di0OHTqEQ4cO4Yc//CF+/OMfS4XKunXrsH79eqSmpqKoqAh6vR6zZ89GY2OjtIzExERkZGQgPT0dubm5aGpqQmxsLDo7O6WY+Ph4lJSUwGAwwGAwoKSkBAkJCU7aZCIiIvIEvnKCH3jgAZv/v/DCC9i8eTMKCgpw5513YuPGjVizZg3mz58PANixYwdCQkKwe/duLF68GGazGdu2bcPOnTsxa9YsAEBaWhrCwsKQnZ2NOXPm4OTJkzAYDCgoKEBUVBQAYOvWrYiOjkZ5eTkmTpzojO2mfrhqCLmPjw9n7CUiIqcadJ+Yzs5OpKeno7m5GdHR0aioqIDRaERMTIwU4+/vj+nTpyMvLw8AUFxcjPb2dpuY0NBQRERESDH5+fnQ6XRSAQMAU6dOhU6nk2L6YrFY0NDQYPNwhWZrh0uW6+kuXmtBWsElpdMgIiIPIruIKS0txZe+9CX4+/tjyZIlyMjIwJ133gmj0QgACAkJsYkPCQmRXjMajRgxYgTGjh1rNyY4OLjXeoODg6WYvqSkpEh9aHQ6HcLCwuRumkPMre0uWa6nu9pkQfbJGqXTICIiDyK7iJk4cSJKSkpQUFCAX/7yl3j88cdx4sQXdyPuOaGZEGLASc56xvQVP9ByVq9eDbPZLD0qKysd3SQiIiLSINlFzIgRI/D1r38dU6ZMQUpKCu6++2789a9/hV6vB4BerSW1tbVS64xer4fVaoXJZLIbU1PT+4y9rq6uVyvPzfz9/aVRU90P0r6GNl6+IyKivg15nhghBCwWCyZMmAC9Xo+srCzpNavVipycHEybNg0AEBkZCT8/P5uY6upqlJWVSTHR0dEwm804ePCgFFNYWAiz2SzFKIk3gHSvl/aUK50CERGplKzRSc899xzuv/9+hIWFobGxEenp6fjkk09gMBjg4+ODxMREJCcnIzw8HOHh4UhOTsbo0aMRHx8PANDpdFi4cCFWrFiBcePGISgoCCtXrsTkyZOl0UqTJk3C3LlzsWjRImzZsgUA8OSTTyI2NpYjk4iIiEgiq4ipqalBQkICqqurodPpcNddd8FgMGD27NkAgFWrVqG1tRVLly6FyWRCVFQUMjMzERAQIC1jw4YN8PX1xYIFC9Da2oqZM2di+/btGD58uBSza9cuLF++XBrFFBcXh9TUVGdsLxEREXkIH+Ghk3c0NDRAp9PBbDY7tX/Mw2/ko+D8dactT23Cgkah8nqr0mnYuLB2Hu74zYdKp0FERH24sHaeU5cn5/eb904iIiIiTWIRIxM79hIREakDixgiIiLSJBYxMrnq3kJEREQkD4sYIiIi0iQWMTJ55liuL3j69hERkedgEUM2qkzqGl5NRETUHxYxREREpEksYoiIiEiTWMQQERGRJrGIISIiIk1iEUNERESaxCJGJo5AJiIiUgcWMURERKRJLGKIiIhIk1jEEBERkSaxiCEiIiJNYhFDREREmsQihlRv7cenlE6BiIhUiEUMqd7rOeeUToGIiFSIRQwRERFpEosYIiIi0iQWMXJxyl4iIiJVYBEjk2AVQ0REpAosYoiIiEiTWMTIJNgQQ0REpAosYoiIiEiTWMQQERGRJrGIISIiIk1iEUNERESaxCKGiIiINElWEZOSkoJ7770XAQEBCA4OxoMPPojy8nKbGCEEkpKSEBoailGjRmHGjBk4fvy4TYzFYsGyZcswfvx4jBkzBnFxcaiqqrKJMZlMSEhIgE6ng06nQ0JCAurr6we3lURERORxZBUxOTk5eOqpp1BQUICsrCx0dHQgJiYGzc3NUsy6deuwfv16pKamoqioCHq9HrNnz0ZjY6MUk5iYiIyMDKSnpyM3NxdNTU2IjY1FZ2enFBMfH4+SkhIYDAYYDAaUlJQgISHBCZs8NBxhTUREpA4+Qgx+5pO6ujoEBwcjJycH3//+9yGEQGhoKBITE/HrX/8awI1Wl5CQELz44otYvHgxzGYzbrnlFuzcuRMPPfQQAODKlSsICwvDRx99hDlz5uDkyZO48847UVBQgKioKABAQUEBoqOjcerUKUycOHHA3BoaGqDT6WA2mxEYGDjYTezlp5vzUHzR5LTlERERadmFtfOcujw5v99D6hNjNpsBAEFBQQCAiooKGI1GxMTESDH+/v6YPn068vLyAADFxcVob2+3iQkNDUVERIQUk5+fD51OJxUwADB16lTodDoppieLxYKGhgabhysMoeYjIiIiJxp0ESOEwLPPPov77rsPERERAACj0QgACAkJsYkNCQmRXjMajRgxYgTGjh1rNyY4OLjXOoODg6WYnlJSUqT+MzqdDmFhYYPdNCIiItKAQRcxTz/9NI4dO4a///3vvV7z8fGx+b8QotdzPfWM6Sve3nJWr14Ns9ksPSorKx3ZDNkG2g4iIiJyj0EVMcuWLcP777+P/fv347bbbpOe1+v1ANCrtaS2tlZqndHr9bBarTCZTHZjampqeq23rq6uVytPN39/fwQGBto8iIiIyHPJKmKEEHj66afx7rvvYt++fZgwYYLN6xMmTIBer0dWVpb0nNVqRU5ODqZNmwYAiIyMhJ+fn01MdXU1ysrKpJjo6GiYzWYcPHhQiiksLITZbJZilMI+MUREROrgKyf4qaeewu7du/Hvf/8bAQEBUouLTqfDqFGj4OPjg8TERCQnJyM8PBzh4eFITk7G6NGjER8fL8UuXLgQK1aswLhx4xAUFISVK1di8uTJmDVrFgBg0qRJmDt3LhYtWoQtW7YAAJ588knExsY6NDKJiIiIPJ+sImbz5s0AgBkzZtg8/9Zbb+HnP/85AGDVqlVobW3F0qVLYTKZEBUVhczMTAQEBEjxGzZsgK+vLxYsWIDW1lbMnDkT27dvx/Dhw6WYXbt2Yfny5dIopri4OKSmpg5mG4mIiMgDDWmeGDVz1Twx81/7DIcv1TtteURERFqm2XlivJFHVnxEREQaxCKGiIiINIlFDBEREWkSixiZPLMHERERkfawiCEiIiJNYhFDREREmsQihoiIiDSJRQwRERFpEosYIiIi0iQWMURERKRJLGKIiIhIk1jEyMRpYoiIiNSBRQwRERFpEosYIiIi0iQWMURERKRJLGKIiIhIk1jEEBERkSaxiCEiIiJNYhFDREREmsQiRi7BmWKIiIjUgEUMERERaRKLGCIiItIkFjFERESkSSxiZGKPGCIiInVgEUNERESaxCKGiIiINIlFDBEREWkSixgiIiLSJBYxREREpEksYoiIiEiTWMQQERGRJskuYg4cOIAHHngAoaGh8PHxwXvvvWfzuhACSUlJCA0NxahRozBjxgwcP37cJsZisWDZsmUYP348xowZg7i4OFRVVdnEmEwmJCQkQKfTQafTISEhAfX19bI3kIiIiDyT7CKmubkZd999N1JTU/t8fd26dVi/fj1SU1NRVFQEvV6P2bNno7GxUYpJTExERkYG0tPTkZubi6amJsTGxqKzs1OKiY+PR0lJCQwGAwwGA0pKSpCQkDCITSQiIiJP5Cv3Dffffz/uv//+Pl8TQmDjxo1Ys2YN5s+fDwDYsWMHQkJCsHv3bixevBhmsxnbtm3Dzp07MWvWLABAWloawsLCkJ2djTlz5uDkyZMwGAwoKChAVFQUAGDr1q2Ijo5GeXk5Jk6cONjtJSIiIg/h1D4xFRUVMBqNiImJkZ7z9/fH9OnTkZeXBwAoLi5Ge3u7TUxoaCgiIiKkmPz8fOh0OqmAAYCpU6dCp9NJMT1ZLBY0NDTYPIiIiMhzObWIMRqNAICQkBCb50NCQqTXjEYjRowYgbFjx9qNCQ4O7rX84OBgKaanlJQUqf+MTqdDWFjYkLenL4I3TyIiIlIFl4xO8vHxsfm/EKLXcz31jOkr3t5yVq9eDbPZLD0qKysHkfnABG8BSUREpApOLWL0ej0A9Gotqa2tlVpn9Ho9rFYrTCaT3Ziamppey6+rq+vVytPN398fgYGBNg8iIiLyXE4tYiZMmAC9Xo+srCzpOavVipycHEybNg0AEBkZCT8/P5uY6upqlJWVSTHR0dEwm804ePCgFFNYWAiz2SzFKIWXk4iIiNRB9uikpqYmnD17Vvp/RUUFSkpKEBQUhK985StITExEcnIywsPDER4ejuTkZIwePRrx8fEAAJ1Oh4ULF2LFihUYN24cgoKCsHLlSkyePFkarTRp0iTMnTsXixYtwpYtWwAATz75JGJjYzkyiYiIiAAMoog5dOgQfvCDH0j/f/bZZwEAjz/+OLZv345Vq1ahtbUVS5cuhclkQlRUFDIzMxEQECC9Z8OGDfD19cWCBQvQ2tqKmTNnYvv27Rg+fLgUs2vXLixfvlwaxRQXF9fv3DRERETkfXyE8MwLJA0NDdDpdDCbzU7tHzPvlU9x/AqHbxMREQHAhbXznLo8Ob/fvHeSTJ5Z8hEREWkPixiZWMMQERGpA4sYIiIi0iQWMURERKRJLGJksj/vMBEREbkLixgiIiLSJBYxREREpEksYmTi6CQiIiJ1YBFDREREmsQihoiIiDSJRQwRERFpEosYIiIi0iQWMURERKRJLGKIiIhIk1jEyCR4G2siIiJVYBFDREREmsQihoiIiDSJRQwRERFpEosYIiIi0iQWMURERKRJLGKIiIhIk1jEyBTzTb3SKRARERFYxMg2ym+40ikQERERWMQQERGRRrGIISIiIk1iEUNERESaxCJGprb2TqVTICIiIrCIka2w4prSKRARERFYxMg2zMdH6RSIiIgILGJkYxFDRESkDixiZGINQ0REpA6qL2Jee+01TJgwASNHjkRkZCQ+/fRTpVMiIiIiFVB1EfOPf/wDiYmJWLNmDY4cOYLvfe97uP/++3Hp0iXFchJCsVUTERHRTVRdxKxfvx4LFy7EL37xC0yaNAkbN25EWFgYNm/erHRqREREpDDVFjFWqxXFxcWIiYmxeT4mJgZ5eXm94i0WCxoaGmweriDAphgiIiI1UG0Rc/XqVXR2diIkJMTm+ZCQEBiNxl7xKSkp0Ol00iMsLMwleS2Y4prlEhERkTyqLWK6+fQYDiSE6PUcAKxevRpms1l6VFZWuiSfH3/ry/jaLWNcsmwiIiIt+fiZ7ym6fl9F127H+PHjMXz48F6tLrW1tb1aZwDA398f/v7+bslt74oZblkPERER9U+1LTEjRoxAZGQksrKybJ7PysrCtGnTFMqKiIiI1EK1LTEA8OyzzyIhIQFTpkxBdHQ03njjDVy6dAlLlixROjUiIiJSmKqLmIceegjXrl3D888/j+rqakREROCjjz7C7bffrnRqREREpDAfITxz+raGhgbodDqYzWYEBgYqnQ4RERE5QM7vt2r7xBARERHZwyKGiIiINIlFDBEREWkSixgiIiLSJBYxREREpEksYoiIiEiTWMQQERGRJrGIISIiIk1iEUNERESapOrbDgxF90TEDQ0NCmdCREREjur+3XbkhgIeW8Q0NjYCAMLCwhTOhIiIiORqbGyETqezG+Ox907q6urClStXEBAQAB8fH6cuu6GhAWFhYaisrOR9mVSG+0bduH/UjftH3bxl/wgh0NjYiNDQUAwbZr/Xi8e2xAwbNgy33XabS9cRGBjo0R8kLeO+UTfuH3Xj/lE3b9g/A7XAdGPHXiIiItIkFjFERESkSSxiBsHf3x9/+MMf4O/vr3Qq1AP3jbpx/6gb94+6cf/05rEde4mIiMizsSWGiIiINIlFDBEREWkSixgiIiLSJBYxREREpEksYmR67bXXMGHCBIwcORKRkZH49NNPlU5JUw4cOIAHHngAoaGh8PHxwXvvvWfzuhACSUlJCA0NxahRozBjxgwcP37cJsZisWDZsmUYP348xowZg7i4OFRVVdnEmEwmJCQkQKfTQafTISEhAfX19TYxly5dwgMPPIAxY8Zg/PjxWL58OaxWq01MaWkppk+fjlGjRuHLX/4ynn/+eYfu56FVKSkpuPfeexEQEIDg4GA8+OCDKC8vt4nhPlLG5s2bcdddd0kTnUVHR+Pjjz+WXud+UZeUlBT4+PggMTFReo77yAUEOSw9PV34+fmJrVu3ihMnTohnnnlGjBkzRly8eFHp1DTjo48+EmvWrBHvvPOOACAyMjJsXl+7dq0ICAgQ77zzjigtLRUPPfSQuPXWW0VDQ4MUs2TJEvHlL39ZZGVlicOHD4sf/OAH4u677xYdHR1SzNy5c0VERITIy8sTeXl5IiIiQsTGxkqvd3R0iIiICPGDH/xAHD58WGRlZYnQ0FDx9NNPSzFms1mEhISIhx9+WJSWlop33nlHBAQEiL/85S+u+wMpbM6cOeKtt94SZWVloqSkRMybN0985StfEU1NTVIM95Ey3n//ffHhhx+K8vJyUV5eLp577jnh5+cnysrKhBDcL2py8OBBcccdd4i77rpLPPPMM9Lz3EfOxyJGhu985ztiyZIlNs994xvfEL/5zW8UykjbehYxXV1dQq/Xi7Vr10rPtbW1CZ1OJ15//XUhhBD19fXCz89PpKenSzGXL18Ww4YNEwaDQQghxIkTJwQAUVBQIMXk5+cLAOLUqVNCiBvF1LBhw8Tly5elmL///e/C399fmM1mIYQQr732mtDpdKKtrU2KSUlJEaGhoaKrq8uJfwn1qq2tFQBETk6OEIL7SG3Gjh0r3nzzTe4XFWlsbBTh4eEiKytLTJ8+XSpiuI9cg5eTHGS1WlFcXIyYmBib52NiYpCXl6dQVp6loqICRqPR5m/s7++P6dOnS3/j4uJitLe328SEhoYiIiJCisnPz4dOp0NUVJQUM3XqVOh0OpuYiIgIhIaGSjFz5syBxWJBcXGxFDN9+nSbiaXmzJmDK1eu4MKFC87/A6iQ2WwGAAQFBQHgPlKLzs5OpKeno7m5GdHR0dwvKvLUU09h3rx5mDVrls3z3EeuwSLGQVevXkVnZydCQkJsng8JCYHRaFQoK8/S/Xe09zc2Go0YMWIExo4dazcmODi41/KDg4NtYnquZ+zYsRgxYoTdmO7/e8M+F0Lg2WefxX333YeIiAgA3EdKKy0txZe+9CX4+/tjyZIlyMjIwJ133sn9ohLp6ek4fPgwUlJSer3GfeQaHnsXa1fx8fGx+b8QotdzNDSD+Rv3jOkr3hkx4vNOb96wz59++mkcO3YMubm5vV7jPlLGxIkTUVJSgvr6erzzzjt4/PHHkZOTI73O/aKcyspKPPPMM8jMzMTIkSP7jeM+ci62xDho/PjxGD58eK8Ktba2tlc1S4Oj1+sB9D4LuPlvrNfrYbVaYTKZ7MbU1NT0Wn5dXZ1NTM/1mEwmtLe3242pra0F0PtsytMsW7YM77//Pvbv34/bbrtNep77SFkjRozA17/+dUyZMgUpKSm4++678de//pX7RQWKi4tRW1uLyMhI+Pr6wtfXFzk5OXjllVfg6+vbbysH99HQsIhx0IgRIxAZGYmsrCyb57OysjBt2jSFsvIsEyZMgF6vt/kbW61W5OTkSH/jyMhI+Pn52cRUV1ejrKxMiomOjobZbMbBgwelmMLCQpjNZpuYsrIyVFdXSzGZmZnw9/dHZGSkFHPgwAGbYYmZmZkIDQ3FHXfc4fw/gAoIIfD000/j3Xffxb59+zBhwgSb17mP1EUIAYvFwv2iAjNnzkRpaSlKSkqkx5QpU/Doo4+ipKQEX/3qV7mPXMF9fYi1r3uI9bZt28SJEydEYmKiGDNmjLhw4YLSqWlGY2OjOHLkiDhy5IgAINavXy+OHDkiDVNfu3at0Ol04t133xWlpaXikUce6XMI4m233Says7PF4cOHxQ9/+MM+hyDeddddIj8/X+Tn54vJkyf3OQRx5syZ4vDhwyI7O1vcdtttNkMQ6+vrRUhIiHjkkUdEaWmpePfdd0VgYKDmhiDK8ctf/lLodDrxySefiOrqaunR0tIixXAfKWP16tXiwIEDoqKiQhw7dkw899xzYtiwYSIzM1MIwf2iRjePThKC+8gVWMTI9Oqrr4rbb79djBgxQnz729+Whp6SY/bv3y8A9Ho8/vjjQogbwxD/8Ic/CL1eL/z9/cX3v/99UVpaarOM1tZW8fTTT4ugoCAxatQoERsbKy5dumQTc+3aNfHoo4+KgIAAERAQIB599FFhMplsYi5evCjmzZsnRo0aJYKCgsTTTz9tM9xQCCGOHTsmvve97wl/f3+h1+tFUlKSpoYfytXXvgEg3nrrLSmG+0gZTzzxhHTsueWWW8TMmTOlAkYI7hc16lnEcB85n48QWpuej4iIiIh9YoiIiEijWMQQERGRJrGIISIiIk1iEUNERESaxCKGiIiINIlFDBEREWkSixgiIiLSJBYxREREpEksYoiIiEiTWMQQERGRJrGIISIiIk1iEUNERESa9P8DbQGEnZOXD+cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df['review_len'] = df.review.map(len)\n",
    "df['review_len'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('data/steam_cleaned.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df = pd.read_csv('data/steam_cleaned.csv') < this is somehow broken :( error in tokenizer later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date_posted</th>\n",
       "      <th>funny</th>\n",
       "      <th>helpful</th>\n",
       "      <th>hour_played</th>\n",
       "      <th>is_early_access_review</th>\n",
       "      <th>recommendation</th>\n",
       "      <th>review</th>\n",
       "      <th>title</th>\n",
       "      <th>labels</th>\n",
       "      <th>review_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-10</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>578</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Played as German Reich Declare war on Belgium...</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "      <td>1</td>\n",
       "      <td>180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>184</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>yes.</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-07</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>892</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Very good game although a bit overpriced in my...</td>\n",
       "      <td>Expansion - Hearts of Iron IV: Man the Guns</td>\n",
       "      <td>1</td>\n",
       "      <td>171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-06-14</td>\n",
       "      <td>126</td>\n",
       "      <td>1086</td>\n",
       "      <td>676</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Out of all the reviews I wrote This one is pro...</td>\n",
       "      <td>Dead by Daylight</td>\n",
       "      <td>1</td>\n",
       "      <td>2094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-06-20</td>\n",
       "      <td>85</td>\n",
       "      <td>2139</td>\n",
       "      <td>612</td>\n",
       "      <td>False</td>\n",
       "      <td>Recommended</td>\n",
       "      <td>Disclaimer I survivor main. I play games for f...</td>\n",
       "      <td>Dead by Daylight</td>\n",
       "      <td>1</td>\n",
       "      <td>1397</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  date_posted  funny  helpful  hour_played  is_early_access_review  \\\n",
       "0  2019-02-10      2        4          578                   False   \n",
       "1  2019-02-10      0        0          184                   False   \n",
       "2  2019-02-07      0        0          892                   False   \n",
       "3  2018-06-14    126     1086          676                   False   \n",
       "4  2017-06-20     85     2139          612                   False   \n",
       "\n",
       "  recommendation                                             review  \\\n",
       "0    Recommended   Played as German Reich Declare war on Belgium...   \n",
       "1    Recommended                                               yes.   \n",
       "2    Recommended  Very good game although a bit overpriced in my...   \n",
       "3    Recommended  Out of all the reviews I wrote This one is pro...   \n",
       "4    Recommended  Disclaimer I survivor main. I play games for f...   \n",
       "\n",
       "                                         title  labels  review_len  \n",
       "0  Expansion - Hearts of Iron IV: Man the Guns       1         180  \n",
       "1  Expansion - Hearts of Iron IV: Man the Guns       1           4  \n",
       "2  Expansion - Hearts of Iron IV: Man the Guns       1         171  \n",
       "3                             Dead by Daylight       1        2094  \n",
       "4                             Dead by Daylight       1        1397  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = df.sample(frac=0.8,random_state=0)\n",
    "test_df = df.drop(train_df.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now we can set our max and move along!\n",
    "\n",
    "Depending on your computer, you might want to start running the next few cells and go make some tea, drink some water, do 10 minutes of yoga/stretches or get some fresh air. It will take a minute to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ds = Dataset.from_pandas(train_df[['review', 'labels']])\n",
    "test_ds = Dataset.from_pandas(test_df[['review', 'labels']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd89fd86106046e58a98571c1fe30508",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/348 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "SEQ_LENGTH=256\n",
    "\n",
    "def tokenization(example):\n",
    "    return tokenizer(example['review'], padding='max_length', truncation=True, max_length=SEQ_LENGTH)\n",
    "\n",
    "train_tok = train_ds.map(tokenization, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b508f039c7d43d28b7adb0ef7dca7c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/87 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_tok = test_ds.map(tokenization, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_tok.set_format(type=\"torch\", columns=[\"input_ids\", \"token_type_ids\", \"attention_mask\", \"labels\"])\n",
    "test_tok.set_format(type=\"torch\", columns=[\"input_ids\", \"token_type_ids\", \"attention_mask\", \"labels\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_tok = train_tok.remove_columns(['review'])\n",
    "test_tok = test_tok.remove_columns(['review'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32\n",
    "MAX_PHYSICAL_BATCH_SIZE = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataloader = DataLoader(train_tok, batch_size=BATCH_SIZE)\n",
    "test_dataloader = DataLoader(test_tok, sampler=SequentialSampler(test_tok), batch_size=BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"mps\") #CHANGE THIS TO FIT YOUR DEVICE PLEASE :D\n",
    "model = model.to(device)\n",
    "\n",
    "# Set the model to train mode (HuggingFace models load in eval mode)\n",
    "model = model.train()\n",
    "# Define optimizer\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPOCHS = 3\n",
    "LOGGING_INTERVAL = 500 # once every how many steps we run evaluation cycle and report metrics\n",
    "EPSILON = 7.5\n",
    "DELTA = 1 / len(train_dataloader) # Parameter for privacy accounting. Probability of not achieving privacy guarantees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(preds, labels):\n",
    "    return (preds == labels).mean()\n",
    "\n",
    "# define evaluation cycle\n",
    "def evaluate(model):    \n",
    "    model.eval()\n",
    "\n",
    "    loss_arr = []\n",
    "    accuracy_arr = []\n",
    "    \n",
    "    for batch in test_dataloader:\n",
    "\n",
    "        with torch.no_grad():\n",
    "            \n",
    "            inputs = {'input_ids':      batch['input_ids'].to(device),\n",
    "                      'attention_mask': batch['attention_mask'].to(device),\n",
    "                      'token_type_ids': batch['token_type_ids'].to(device),\n",
    "                      'labels':         batch['labels'].to(device)}\n",
    "\n",
    "\n",
    "            outputs = model(**inputs)\n",
    "            loss, logits = outputs[:2]\n",
    "            \n",
    "            preds = np.argmax(logits.detach().cpu().numpy(), axis=1)\n",
    "            labels = batch['labels'].detach().cpu().numpy()\n",
    "            \n",
    "            loss_arr.append(loss.item())\n",
    "            accuracy_arr.append(accuracy(preds, labels))\n",
    "    \n",
    "    model.train()\n",
    "    return np.mean(loss_arr), np.mean(accuracy_arr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/py3data/lib/python3.8/site-packages/opacus/privacy_engine.py:142: UserWarning: Secure RNG turned off. This is perfectly fine for experimentation as it allows for much faster training performance, but remember to turn it on and retrain one last time before production with ``secure_mode`` turned on.\n",
      "  warnings.warn(\n",
      "/opt/anaconda3/envs/py3data/lib/python3.8/site-packages/opacus/accountants/analysis/rdp.py:332: UserWarning: Optimal order is the largest alpha. Please consider expanding the range of alphas to get a tighter privacy bound.\n",
      "  warnings.warn(\n",
      "/opt/anaconda3/envs/py3data/lib/python3.8/site-packages/opacus/accountants/analysis/prv/prvs.py:50: RuntimeWarning: invalid value encountered in log\n",
      "  z = np.log((np.exp(t) + q - 1) / q)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "MAX_GRAD_NORM = 0.1\n",
    "\n",
    "privacy_engine = PrivacyEngine()\n",
    "\n",
    "model, optimizer, train_dataloader = privacy_engine.make_private_with_epsilon(\n",
    "    module=model,\n",
    "    optimizer=optimizer,\n",
    "    data_loader=train_dataloader,\n",
    "    target_delta=DELTA,\n",
    "    target_epsilon=EPSILON, \n",
    "    epochs=EPOCHS,\n",
    "    max_grad_norm=MAX_GRAD_NORM,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4219e45ee4d40c782711fef6cb313b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10872 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/py3data/lib/python3.8/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.\n",
      "  warnings.warn(\"Using a non-full backward hook when the forward contains multiple autograd Nodes \"\n",
      "/opt/anaconda3/envs/py3data/lib/python3.8/site-packages/torch/nn/modules/module.py:1078: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.\n",
      "  warnings.warn(\"Using non-full backward hooks on a Module that does not return a \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 | Step: 500 | Train loss: 1.971 | Eval loss: 2.042 | Eval accuracy: 0.698 | ɛ: 2.68\n",
      "Epoch: 1 | Step: 1000 | Train loss: 1.970 | Eval loss: 2.018 | Eval accuracy: 0.698 | ɛ: 3.30\n",
      "Epoch: 1 | Step: 1500 | Train loss: 1.928 | Eval loss: 1.723 | Eval accuracy: 0.698 | ɛ: 3.68\n",
      "Epoch: 1 | Step: 2000 | Train loss: 1.834 | Eval loss: 1.450 | Eval accuracy: 0.720 | ɛ: 3.94\n",
      "Epoch: 1 | Step: 2500 | Train loss: 1.711 | Eval loss: 1.144 | Eval accuracy: 0.785 | ɛ: 4.14\n",
      "Epoch: 1 | Step: 3000 | Train loss: 1.628 | Eval loss: 1.137 | Eval accuracy: 0.794 | ɛ: 4.30\n",
      "Epoch: 1 | Step: 3500 | Train loss: 1.561 | Eval loss: 1.089 | Eval accuracy: 0.804 | ɛ: 4.44\n",
      "Epoch: 1 | Step: 4000 | Train loss: 1.518 | Eval loss: 1.106 | Eval accuracy: 0.807 | ɛ: 4.56\n",
      "Epoch: 1 | Step: 4500 | Train loss: 1.484 | Eval loss: 1.213 | Eval accuracy: 0.803 | ɛ: 4.66\n",
      "Epoch: 1 | Step: 5000 | Train loss: 1.452 | Eval loss: 1.097 | Eval accuracy: 0.813 | ɛ: 4.76\n",
      "Epoch: 1 | Step: 5500 | Train loss: 1.426 | Eval loss: 1.122 | Eval accuracy: 0.814 | ɛ: 4.84\n",
      "Epoch: 1 | Step: 6000 | Train loss: 1.402 | Eval loss: 1.083 | Eval accuracy: 0.819 | ɛ: 4.92\n",
      "Epoch: 1 | Step: 6500 | Train loss: 1.381 | Eval loss: 1.346 | Eval accuracy: 0.806 | ɛ: 5.00\n",
      "Epoch: 1 | Step: 7000 | Train loss: 1.362 | Eval loss: 1.054 | Eval accuracy: 0.824 | ɛ: 5.06\n",
      "Epoch: 1 | Step: 7500 | Train loss: 1.345 | Eval loss: 1.061 | Eval accuracy: 0.827 | ɛ: 5.13\n",
      "Epoch: 1 | Step: 8000 | Train loss: 1.332 | Eval loss: 1.108 | Eval accuracy: 0.826 | ɛ: 5.19\n",
      "Epoch: 1 | Step: 8500 | Train loss: 1.320 | Eval loss: 1.079 | Eval accuracy: 0.827 | ɛ: 5.25\n",
      "Epoch: 1 | Step: 9000 | Train loss: 1.309 | Eval loss: 1.089 | Eval accuracy: 0.827 | ɛ: 5.31\n",
      "Epoch: 1 | Step: 9500 | Train loss: 1.299 | Eval loss: 1.080 | Eval accuracy: 0.829 | ɛ: 5.36\n",
      "Epoch: 1 | Step: 10000 | Train loss: 1.290 | Eval loss: 1.215 | Eval accuracy: 0.822 | ɛ: 5.41\n",
      "Epoch: 1 | Step: 10500 | Train loss: 1.281 | Eval loss: 1.168 | Eval accuracy: 0.828 | ɛ: 5.46\n",
      "Epoch: 1 | Step: 11000 | Train loss: 1.274 | Eval loss: 1.140 | Eval accuracy: 0.827 | ɛ: 5.51\n",
      "Epoch: 1 | Step: 11500 | Train loss: 1.268 | Eval loss: 1.091 | Eval accuracy: 0.830 | ɛ: 5.55\n",
      "Epoch: 1 | Step: 12000 | Train loss: 1.261 | Eval loss: 1.079 | Eval accuracy: 0.830 | ɛ: 5.60\n",
      "Epoch: 1 | Step: 12500 | Train loss: 1.254 | Eval loss: 1.183 | Eval accuracy: 0.824 | ɛ: 5.64\n",
      "Epoch: 1 | Step: 13000 | Train loss: 1.248 | Eval loss: 1.094 | Eval accuracy: 0.832 | ɛ: 5.68\n",
      "Epoch: 1 | Step: 13500 | Train loss: 1.243 | Eval loss: 1.147 | Eval accuracy: 0.829 | ɛ: 5.72\n",
      "Epoch: 1 | Step: 14000 | Train loss: 1.238 | Eval loss: 1.104 | Eval accuracy: 0.830 | ɛ: 5.76\n",
      "Epoch: 1 | Step: 14500 | Train loss: 1.233 | Eval loss: 1.096 | Eval accuracy: 0.833 | ɛ: 5.80\n",
      "Epoch: 1 | Step: 15000 | Train loss: 1.228 | Eval loss: 1.133 | Eval accuracy: 0.830 | ɛ: 5.84\n",
      "Epoch: 1 | Step: 15500 | Train loss: 1.223 | Eval loss: 1.148 | Eval accuracy: 0.832 | ɛ: 5.88\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35eff8a4aeb845ef87af4df8bfedecd9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10872 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 2 | Step: 500 | Train loss: 1.100 | Eval loss: 1.142 | Eval accuracy: 0.833 | ɛ: 5.93\n",
      "Epoch: 2 | Step: 1000 | Train loss: 1.123 | Eval loss: 1.104 | Eval accuracy: 0.834 | ɛ: 5.97\n",
      "Epoch: 2 | Step: 1500 | Train loss: 1.116 | Eval loss: 1.151 | Eval accuracy: 0.831 | ɛ: 6.00\n",
      "Epoch: 2 | Step: 2000 | Train loss: 1.116 | Eval loss: 1.103 | Eval accuracy: 0.834 | ɛ: 6.04\n",
      "Epoch: 2 | Step: 2500 | Train loss: 1.115 | Eval loss: 1.112 | Eval accuracy: 0.835 | ɛ: 6.07\n",
      "Epoch: 2 | Step: 3000 | Train loss: 1.120 | Eval loss: 1.103 | Eval accuracy: 0.835 | ɛ: 6.10\n",
      "Epoch: 2 | Step: 3500 | Train loss: 1.123 | Eval loss: 1.131 | Eval accuracy: 0.833 | ɛ: 6.14\n",
      "Epoch: 2 | Step: 4000 | Train loss: 1.122 | Eval loss: 1.188 | Eval accuracy: 0.830 | ɛ: 6.17\n",
      "Epoch: 2 | Step: 4500 | Train loss: 1.122 | Eval loss: 1.128 | Eval accuracy: 0.832 | ɛ: 6.20\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(1, EPOCHS+1):\n",
    "    losses = []\n",
    "\n",
    "    with BatchMemoryManager(\n",
    "        data_loader=train_dataloader, \n",
    "        max_physical_batch_size=MAX_PHYSICAL_BATCH_SIZE, \n",
    "        optimizer=optimizer\n",
    "    ) as memory_safe_data_loader:\n",
    "        for step, batch in enumerate(tqdm(memory_safe_data_loader)):\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            inputs = {'input_ids':      batch['input_ids'].to(device),\n",
    "                      'attention_mask': batch['attention_mask'].to(device),\n",
    "                      'token_type_ids': batch['token_type_ids'].to(device),\n",
    "                      'labels':         batch['labels'].to(device)}\n",
    "\n",
    "            outputs = model(**inputs) # output = loss, logits, hidden_states, attentions\n",
    "            \n",
    "            loss = outputs[0]\n",
    "            loss.backward()\n",
    "            losses.append(loss.item())\n",
    "\n",
    "            optimizer.step()\n",
    "\n",
    "            if step > 0 and step % LOGGING_INTERVAL == 0:\n",
    "                train_loss = np.mean(losses)\n",
    "                eps = privacy_engine.get_epsilon(DELTA)\n",
    "\n",
    "                eval_loss, eval_accuracy = evaluate(model)\n",
    "\n",
    "                print(\n",
    "                  f\"Epoch: {epoch} | \"\n",
    "                  f\"Step: {step} | \"\n",
    "                  f\"Train loss: {train_loss:.3f} | \"\n",
    "                  f\"Eval loss: {eval_loss:.3f} | \"\n",
    "                  f\"Eval accuracy: {eval_accuracy:.3f} | \"\n",
    "                  f\"ɛ: {eps:.2f}\"\n",
    "                )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenges\n",
    "\n",
    "- Play with some of the Opacus privacy engine settings and see if your performance changes. Can you find a better information v. utility balance?\n",
    "- Walk through [an Opacus tutorial](https://opacus.ai/tutorials/building_image_classifier) or try out one from [tf-privacy](https://github.com/tensorflow/privacy/blob/master/tutorials/walkthrough/README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}