optimal-experimental-design/illustrate-active-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "postal-arlington",
   "metadata": {},
   "source": [
    "# Active Learning over Molecular Space\n",
    "This notebook is going to show how to perform active learning to find molecules with large bandgaps.\n",
    "We are going to use the [modAL](https://modal-python.readthedocs.io/en/latest/index.html) library to demonstrate a few different active learning algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "foreign-decimal",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:08.359944Z",
     "iopub.status.busy": "2021-02-25T12:47:08.358562Z",
     "iopub.status.idle": "2021-02-25T12:47:11.668151Z",
     "shell.execute_reply": "2021-02-25T12:47:11.667009Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.ensemble import BaggingRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.pipeline import Pipeline\n",
    "from modAL.models import CommitteeRegressor\n",
    "from modAL.acquisition import max_UCB\n",
    "from rdkit.Chem import AllChem, DataStructs\n",
    "from rdkit import Chem\n",
    "from tqdm import tqdm\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "forty-distinction",
   "metadata": {},
   "source": [
    "## Load Example Data\n",
    "Get the QM9 dataset from our molecular property prediction exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "hearing-denial",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:11.677032Z",
     "iopub.status.busy": "2021-02-25T12:47:11.675631Z",
     "iopub.status.idle": "2021-02-25T12:47:15.928137Z",
     "shell.execute_reply": "2021-02-25T12:47:15.927526Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 25000 molecules\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_json('../molecular-property-prediction/datasets/qm9.json.gz', lines=True)\n",
    "print(f'Loaded {len(data)} molecules')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "boxed-happiness",
   "metadata": {},
   "source": [
    "Split off some molecules to use as our initial dataset and search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "medieval-climb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:15.934685Z",
     "iopub.status.busy": "2021-02-25T12:47:15.932371Z",
     "iopub.status.idle": "2021-02-25T12:47:15.943287Z",
     "shell.execute_reply": "2021-02-25T12:47:15.942770Z"
    }
   },
   "outputs": [],
   "source": [
    "seed_data, search_data = train_test_split(data, train_size=8, test_size=8000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pressing-confusion",
   "metadata": {},
   "source": [
    "## Build the Tools Needed for Active Learning\n",
    "We first need to create a model to learn from the data we have already collected and an \"oracle\" that we provide the properties for a molecule on request.\n",
    "\n",
    "### Building a KNN Model\n",
    "We are going to use a simple KNN model based on the fingerprints of a molecule (see the [Chemoinformatics tutorials](../molecular-property-prediction/chemoinformatics/README.md))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "outside-stuart",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:15.951349Z",
     "iopub.status.busy": "2021-02-25T12:47:15.950765Z",
     "iopub.status.idle": "2021-02-25T12:47:15.953878Z",
     "shell.execute_reply": "2021-02-25T12:47:15.953447Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_morgan_fingerprints(smiles: str, fingerprint_length: int, fingerprint_radius: int):\n",
    "    \"\"\"Get Morgan Fingerprint of a specific SMILES string.\n",
    "    Adapted from: <https://github.com/google-research/google-research/blob/\n",
    "    dfac4178ccf521e8d6eae45f7b0a33a6a5b691ee/mol_dqn/chemgraph/dqn/deep_q_networks.py#L750>\n",
    "    Args:\n",
    "      graph (str): The molecule as a SMILES string\n",
    "      fingerprint_length (int): Bit-length of fingerprint\n",
    "      fingerprint_radius (int): Radius used to compute fingerprint\n",
    "    Returns:\n",
    "      np.array. shape = [hparams, fingerprint_length]. The Morgan fingerprint.\n",
    "    \"\"\"\n",
    "    # Parse the molecule\n",
    "    molecule = Chem.MolFromSmiles(smiles)\n",
    "\n",
    "    # Compute the fingerprint\n",
    "    fingerprint = AllChem.GetMorganFingerprintAsBitVect(\n",
    "        molecule, fingerprint_radius, fingerprint_length)\n",
    "    arr = np.zeros((1,), dtype=np.bool)\n",
    "\n",
    "    # ConvertToNumpyArray takes ~ 0.19 ms, while\n",
    "    # np.asarray takes ~ 4.69 ms\n",
    "    DataStructs.ConvertToNumpyArray(fingerprint, arr)\n",
    "    return arr\n",
    "\n",
    "\n",
    "class MorganFingerprintTransformer(BaseEstimator):\n",
    "    \"\"\"Class that converts SMILES strings to fingerprint vectors\"\"\"\n",
    "    \n",
    "    def __init__(self, length: int = 256, radius: int = 4):\n",
    "        self.length = length\n",
    "        self.radius = radius\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self  # Do need to do anything\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        \"\"\"Compute the fingerprints\n",
    "        \n",
    "        Args:\n",
    "            X: List of SMILES strings\n",
    "        Returns:\n",
    "            Array of fingerprints\n",
    "        \"\"\"\n",
    "        fing = [compute_morgan_fingerprints(m, self.length, self.radius) for m in np.squeeze(X)]\n",
    "        return np.vstack(fing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "future-tomorrow",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:15.957585Z",
     "iopub.status.busy": "2021-02-25T12:47:15.957120Z",
     "iopub.status.idle": "2021-02-25T12:47:15.960245Z",
     "shell.execute_reply": "2021-02-25T12:47:15.959799Z"
    }
   },
   "outputs": [],
   "source": [
    "model = Pipeline([\n",
    "    ('fp', MorganFingerprintTransformer()),\n",
    "    ('model', KNeighborsRegressor(metric='jaccard', n_jobs=-1))  # Uses the Tanimoto similarity metric\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "visible-conviction",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:15.988009Z",
     "iopub.status.busy": "2021-02-25T12:47:15.986627Z",
     "iopub.status.idle": "2021-02-25T12:47:15.995502Z",
     "shell.execute_reply": "2021-02-25T12:47:15.995029Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('fp', MorganFingerprintTransformer()),\n",
       "                ('model', KNeighborsRegressor(metric='jaccard', n_jobs=-1))])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(seed_data['smiles_0'].values[:, None], seed_data['bandgap'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "gentle-sussex",
   "metadata": {},
   "source": [
    "We can use [bootstrap aggregation (bagging)](https://en.wikipedia.org/wiki/Bootstrap_aggregating) to get an estimate of the uncertainty of this model by first training an ensemble of models using sklearn's [BaggingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) and then assessing the variance of the models. \n",
    "\n",
    "*NOTE*: Bagging, in short, works by training each model on a different subset of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "opening-twenty",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:15.999192Z",
     "iopub.status.busy": "2021-02-25T12:47:15.998696Z",
     "iopub.status.idle": "2021-02-25T12:47:16.001907Z",
     "shell.execute_reply": "2021-02-25T12:47:16.001364Z"
    }
   },
   "outputs": [],
   "source": [
    "model = BaggingRegressor(model, n_estimators=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "radical-participant",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:16.036723Z",
     "iopub.status.busy": "2021-02-25T12:47:16.036233Z",
     "iopub.status.idle": "2021-02-25T12:47:16.039842Z",
     "shell.execute_reply": "2021-02-25T12:47:16.040226Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingRegressor(base_estimator=Pipeline(steps=[('fp',\n",
       "                                                 MorganFingerprintTransformer()),\n",
       "                                                ('model',\n",
       "                                                 KNeighborsRegressor(metric='jaccard',\n",
       "                                                                     n_jobs=-1))]),\n",
       "                 n_estimators=8)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(seed_data['smiles_0'].values[:, None], seed_data['bandgap'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "answering-moderator",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:16.043648Z",
     "iopub.status.busy": "2021-02-25T12:47:16.043161Z",
     "iopub.status.idle": "2021-02-25T12:47:16.046254Z",
     "shell.execute_reply": "2021-02-25T12:47:16.045866Z"
    }
   },
   "outputs": [],
   "source": [
    "committee = CommitteeRegressor(model.estimators_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "particular-trail",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:16.081057Z",
     "iopub.status.busy": "2021-02-25T12:47:16.070535Z",
     "iopub.status.idle": "2021-02-25T12:47:23.667871Z",
     "shell.execute_reply": "2021-02-25T12:47:23.666809Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred, y_std = committee.predict(search_data['smiles_0'], return_std=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "saving-break",
   "metadata": {},
   "source": [
    "Plot the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "improving-sucking",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:23.710153Z",
     "iopub.status.busy": "2021-02-25T12:47:23.709019Z",
     "iopub.status.idle": "2021-02-25T12:47:24.290550Z",
     "shell.execute_reply": "2021-02-25T12:47:24.289983Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe51b4b9d10>]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzqklEQVR4nO3deXxU1f3/8ddnJpOFkIQlrAGVigohLGLAIopSK4v1IWLd27pUmyLgWhSr4lL0W/e6YSniUu2vUFxQXBFrLSqIBBdkEUHAAhETyEKWyazn98fMhEmYhMl6J5PP8/HgQWbmnpnP3GTec+65954rxhiUUkrFL5vVBSillGpdGvRKKRXnNOiVUirOadArpVSc06BXSqk4l2B1AZFkZmaao446yuoylFIq5lVVVbF161a8Xu8+Y0yPSMvEZNAfddRR5OfnW12GUkrFLGMMIkJ5eTm//vWvWbZs2ff1LatDN0op1Y4YY/j73//OuHHjcLlcpKWl8frrrzfYRoNeKaXaie+//57Jkydz+eWXY4yhuLg4qnYa9EopFeP8fj9PPvkkQ4YM4ZNPPuHJJ59k5cqV9OnTJ6r2UQW9iEwSkS0isk1EbmlguVEi4hOR88Lu2ykiX4vIlyKiA+9KKdVIfr+f5557jpNPPpkNGzYwY8YMbLbo++mH3RkrInZgHnAGsBtYKyLLjDGbIix3P7A8wtOMN8bsi7oqpZTq4DweD48//jiXX3453bt3Z8WKFXTt2hURafRzRfOVMBrYZozZboxxA4uBKRGWuwZ4BShsdBVKKaVqfPHFF4wePZpZs2axePFiALp169akkIfogj4L2BV2e3fwvhoikgVMBeZHaG+A90RknYjk1fciIpInIvkikl9UVBRFWUopFV+cTid//OMfGTVqFHv37uXVV19lxowZzX7eaII+0ldI3bmNHwVmG2N8EZYda4wZCUwGZojIuEgvYoxZYIzJNcbk9ugR8Zh/pZSKazfddBP33Xcfl112GZs2bWLq1Kkt8rzRnDC1G+gfdrsfUFBnmVxgcXCzIhM4U0S8xpjXjDEFAMaYQhFZSmAoaGWzK1dKqThQXl5OeXk5ffv25Y9//CNTpkzhjDPOaNHXiKZHvxY4RkQGiEgicBGwLHwBY8wAY8xRxpijgJeB6caY10QkVUTSAEQkFZgAbGjRd6CUUu3Uu+++S05ODpdeeikAWVlZLR7yEEXQG2O8wEwCR9NsBpYYYzaKyDQRmXaY5r2Aj0XkK+Az4C1jzLvNLVoppdqz/fv3c9lllzF58mRSU1OZO3duq76exOKlBHNzc43OdaOUikfr1q3jzDPPpLi4mFtuuYXbb7+dpKSkZj+viKwzxuRGeiwmJzVTSql4E5qE7Nhjj+Wkk07irrvuYvjw4W3y2joFglJKtSJjDM8++yxjx46lurqatLQ0li5d2mYhDxr0SinVanbs2MGECRO48sorcTgclJaWWlKHBr1SSrUwn8/HY489Rk5ODmvWrOGvf/0r//nPf+jdu7cl9egYvVJKtTBjDC+++CKnnXYa8+fPp3///odv1Iq0R6+UUi3A4/HwwAMPsH//fhISElixYgVvvvmm5SEPGvRKKdVs69atIzc3l9mzZ7NkyRKAJs802Ro06JVSqomcTiezZ89m9OjRFBUV8dprr3H11VdbXdYhNOiVUqqJbrrpJh544AF++9vfsmnTJqZMiTSDu/V0Z6xSSjXCgQMHKC8vJysri1tvvZWpU6dy+umnW11Wg7RHr5RSUXrrrbcYMmQIl112GQB9+/aN+ZAHDXqllDqsffv28etf/5qzzjqLjIwM7r33XqtLahQdulFKqQbk5+dz5plnUlpayp133smtt95KYmKi1WU1iga9UkpFEJqEbNCgQYwbN44777yToUOHWl1Wk+jQjVJKhTHG8PTTT3PSSSdRXV1N586defnll9ttyIMGvVJK1fjuu+84/fTTycvLIzk52bJJyFqaBr1SqsPz+Xw88sgjDB06lHXr1vG3v/2Nf//735ZNQtbSogp6EZkkIltEZJuI3NLAcqNExCci5zW2rVJKWcUYw6JFizj99NPZuHEjeXl52Gzx0w8+7DsRETswD5gMZAMXi0h2PcvdT+Daso1qq5RSbc3tdvPnP/+Zffv21UxCtmzZMvr162d1aS0umq+s0cA2Y8x2Y4wbWAxEOs/3GuAVoLAJbZVSqs189tlnnHDCCdx66628/PLLAHTp0iVmJiFradEEfRawK+z27uB9NUQkC5gKzG9sW6WUaitVVVXMmjWLMWPGUFJSwhtvvMG0adOsLqvVRRP0kb7iTJ3bjwKzjTG+JrQNLCiSJyL5IpJfVFQURVlKKdU4N910Ew8//DB5eXls3LiRs846y+qS2kQ0J0ztBsJnzu8HFNRZJhdYHNzsyQTOFBFvlG0BMMYsABYA5ObmRvwyUEqpxiorK6O8vJx+/fpx2223cf7553PaaadZXVabiqZHvxY4RkQGiEgicBGwLHwBY8wAY8xRxpijgJeB6caY16Jpq5RSreWNN94gOzu71iRkHS3kIYqgN8Z4gZkEjqbZDCwxxmwUkWki0uDgVn1tm1+2UkrVr6ioiEsuuYSzzz6b7t27c99991ldkqXEmNgbJcnNzTX5+flWl6GUaofWrl3L5MmTOXDgAHPmzGH27NntbhKyphCRdcaY3EiP6aRmSqm4EJqEbPDgwfzsZz/jzjvvZMiQIVaXFRPi59QvpVSH5Pf7mT9/PieeeGLNJGRLlizRkA+jQa+Uare2bt3Kz372M66++mrS09MpKyuzuqSYpEGvlGp3vF4vDz74IMOGDePLL7/kmWeeYcWKFfTq1cvq0mKSjtErpdqll156iYkTJ/LUU0/Rt29fq8uJaRr0Sql2weVy8dBDD/H73/+ezMxM3n//fdLS0uJ2fpqWpEM3SqmY9+mnnzJy5Ehuv/12XnnlFQDS09M15KOkQa+UilmVlZXccMMNnHTSSZSXl/P222/z+9//3uqy2h0NeqVUzLr55pt59NFHufrqq9mwYQOTJ0+2uqR2Sc+MVUrFlNLSUsrLy+nfvz8//PADW7duZdy4cVaXFfMaOjNWe/RKqZjx2muvkZ2dzeWXXw5Anz59NORbgAa9UspyP/74IxdccAFTp06lV69ePPjgg1aXFFf08EqllKU+++wzJk+eTEVFBffeey833XQTDofD6rLiiga9UsoSfr8fm83GkCFDmDBhAnfeeSeDBg2yuqy4pEM3Sqk25ff7eeqppxg9ejROp5PU1FQWLVqkId+KNOiVUm3m22+/5bTTTmPGjBl069aN8vJyq0vqEDTolVKtzuv1cv/99zNs2DC+/vprnnvuOZYvX07Pnj2tLq1D0DF6pVSrExFeffVVfvGLXzBv3jx69+5tdUkdSlQ9ehGZJCJbRGSbiNwS4fEpIrJeRL4UkXwROTnssZ0i8nXosZYsXikVu6qrq/nTn/5EUVERdrud999/n1deeUVD3gKH7dGLiB2YB5wB7AbWisgyY8ymsMX+DSwzxhgRGQYsAcL3rIw3xuxrwbqVUjHsk08+4corr2TLli307t2bvLw80tLSrC6rw4qmRz8a2GaM2W6McQOLgSnhCxhjKszBuRRSgdibV0Ep1eoqKiq49tprOeWUU6iurmb58uXk5eVZXVaHF03QZwG7wm7vDt5Xi4hMFZFvgLeA34Y9ZID3RGSdiNT7GxeRvOCwT35RUVF01SulYsrNN9/Mk08+ycyZM9mwYQMTJkywuiRFdEEfacLnQ3rsxpilxphBwDnA3LCHxhpjRgKTgRkiEnHiCmPMAmNMrjEmt0ePHlGUpZSKBSUlJfzvf/8DYM6cOXz00Uc8/vjjdO7c2eLKVEg0Qb8b6B92ux9QUN/CxpiVwNEikhm8XRD8vxBYSmAoSCkVB1555RUGDx5caxKysWPHWluUOkQ0Qb8WOEZEBohIInARsCx8AREZKMFLvYjISCAR2C8iqSKSFrw/FZgAbGjJN6CUant79+7lvPPO47zzzqNv37488sgjVpekGnDYo26MMV4RmQksB+zAs8aYjSIyLfj4fOCXwKUi4gGcwIXBI3B6AUuD3wEJwD+NMe+20ntRSrWBNWvWMHnyZKqqqrjvvvu48cYbdRKyGKcXHlFKRSU0CVllZSV5eXnccccdHHfccVaXpYL0wiNKqSbz+Xw88cQTjBo1qmYSsv/3//6fhnw7okGvlKrX5s2bGTduHNdeey09e/akoqLC6pJUE2jQK6UO4fV6uffeexkxYgTffPMNL7zwAm+//TZ66HP7pJOaKaUOISK8+eabTJkyhSeeeIJevXpZXZJqBg16pRQATqeT+++/nxkzZtCjRw9WrFihJz3FCR26UUrx8ccfM2LECO6++25ef/11AA35OKJBr1QHVl5ezsyZMznllFNwu92sWLGCq666yuqyVAvToFeqA5s9ezZPPfUU119/PRs2bODnP/+51SWpVqAnTCnVwezfv5+KigqOPPJI9u7dy44dOxgzZozVZalm0hOmlFIYY1iyZAmDBw/miiuuAKB3794a8h2ABr1SHUBBQQHnnnsuF154IUcccQSPPvqo1SWpNqSHVyoV5z799FMmTZqEy+XigQce4IYbbiAhQT/6HYn+tpWKUz6fD7vdztChQzn77LOZM2cOxxxzjNVlKQvo0I1Sccbn8/HYY4+Rm5tbMwnZCy+8oCHfgWnQKxVHNm3axMknn8z1119P3759dRIyBWjQKxUXvF4vc+fO5fjjj2fr1q384x//4M0339RJyBSgY/RKxQUR4e233+bcc8/lscceo2fPnlaXpGJIVD16EZkkIltEZJuI3BLh8Skisl5EvhSRfBE5Odq2SqmmqaqqYs6cORQWFmK323n//fdZtGiRhrw6xGGDXkTswDxgMpANXCwi2XUW+zcw3BgzAvgtsLARbZVSjfThhx8yfPhw7rnnHt544w0AUlNTLa5KxapoevSjgW3GmO3GGDewGJgSvoAxpsIcnEshFTDRtlVKRa+srIxp06Yxfvx4jDF88MEHXHnllVaXpWJcNEGfBewKu707eF8tIjJVRL4B3iLQq4+6bbB9XnDYJ7+oqCia2pXqcG655Raefvpp/vCHP7B+/XrGjx9vdUmqHYgm6CXCfYfMhGaMWWqMGQScA8xtTNtg+wXGmFxjTK4eKaDUQfv27WPnzp0A3HXXXaxevZqHHnqITp06WVuYajeiCfrdQP+w2/2AgvoWNsasBI4WkczGtlVKHWSMYfHixQwePJjf/jawkdyrVy9Gjx5tcWWqvYkm6NcCx4jIABFJBC4CloUvICIDRUSCP48EEoH90bRVSh1qz549TJkyhYsvvpif/OQnPP7441aXpNqxwx5Hb4zxishMYDlgB541xmwUkWnBx+cDvwQuFREP4AQuDO6cjdi2ld6LUnFh9erVTJo0CY/Hw8MPP8x1112H3W63uizVjumFR5SKEaFJyKqqqpg+fTq33347AwcOtLos1U7ohUeUimE+n4+HH36YkSNH4nQ66dSpE88//7yGvGoxGvRKWWjDhg2MGTOGWbNmccQRR1BZWWl1SSoOadArZQGPx8Ndd93FyJEj2bFjB4sWLWLZsmVkZmZaXZqKQzpGr+r1Un7gXLfzc/tHfGztzmJGHdXtkMcbahfNa67dWQzQ4HOHC9UR+nntjmIy05J4adpJDbY9P7c/N7/8FTv2VXJB8HXqvvZL+bt46j/bAMhMS2JAZio79lWyr9zF9PEDWbuzmA82F1Ja5Qbg+CO71jz/BWG1h+oCGDWgGyf0z+DGy86hOqk7/SdfTdfu3Smv9pKWnEBRuYtkh53ZkwexJH8XAzIDUxus3VFMebUXgAE9UmtqCRdaFuCDzYUAzJ486JAaMtOSDmkXaX2Hr7cl+btq3ndo3YTWY0N/D6ptNDRGr0EfBxobrIdbPjwQw0M0vE2kD/ZL+btYEtY2FDqhIA1/vdCyAzJTeeC84Ye8ZijAQgEU/tir63Zjtwl9u6RQXu2lvNpDWrKjJhjX7yrFYbcxuG86O4oqqfb4SHYEjlqp9vjw+PwApCU7KK/24PYZUhPteHx+PD6Dwy41j6clJ7BzfxUGSBDwmsBZgNF8apKCz+P2GQzg91RTtupfZJ54Dinp3aiorARHMgLYBfwGOiXaqXT7onr+uq/l8Rn8wdsJwVMVvQY6J9pJdtgpr/bg8xu8JrAp3y01Mfg+AwffZaYlcUFu/1pfLm+v/wGPz09asoPiysCX2Xm5/XjgvOGcP38V+8pdnHN8Fhv2lPHFrlJ6piXx8AXDGdwnI2Kd9f3tNfZ+daiGgl6nKY5Bze0R19c2Um+4vuULSp0s37iXwnIXJZVucrIy2FPqrOnxRnqe+h6DQECHB/uKTXtZ930J674vYcOeslqvW+Hy4g0m1tYfy1m+YS+VLi9dOiUyoEcqPgN+n2HH/qqadq5KNze9vP7gbZ+PtTtLam5XuH2H1OQKBlfdx10+U/P4vrBlvMH0jTaEQ88D4K0qw+88QNqoqXjsSfjcPnAk1zxf6Lkj1dnY1wqvNfScdZ/XDzXvLfT/jv1VrN1Zgg1Yt7MEu03w+ANPZJzumi+Rj7fu47H3v2VfuYu9ZU4WrNxOYoKNSpcHl8fHDf/6imt+djRV7kALDWnraY8+RtTdDA79HM2y4T3r8KGIuo9DIEg37CmjpMpD104ONu4pI6tbJy7I7V+zXEGpk1c/34PT48Xl8dEjLZn0FAcllS72HnDh8fnxGzAGEmxCv67JpCQmUFrppuCAq6ZOoXbv13Bwp5CfjsEYPya4FUDgnEKC5xbGJVvwF+4nsFWR7LDTIy2JUQOCQ2s7ihk1oFvNVtz973yDx+dHBLx+w5ifdOfGCccyuE9Gra2+0N9vvH1ptOQWiw7dtAORhkvCh0Qg8Mew+YcyHnnvW0qqPFS5vTjdXlISE6hy+zimZ2f2lDpxun2MGtCtZiw5My2JkwdmsnrbPr4tqiQ10UbfjBQcdhtf7CrlyO6BOVMKSp0M7JnG13tKcXkNNgLZZJODPbtIbHSc4G6sup+veA75+gjQPdWBx+uj0uPHbrOR2TmRFLvw3X5nrQ5AalICWV1SuOb0o9lWWMlzn+zAEBhmSnEkkNUlmZysDK77+bFt+h5aengUqPVZHjswk0k5veod8oqGDt20E2t3FlNa5WZ3iZPtRZXsLnEyKadXzeOPvf8t//22iE4OO11SEthb5qSw3EWv9CQ6JwV24u3YV4nP76f8Gy+V1R48PsPesiq+K6rkgNMNBiqqhcJyF5mpSTjdPr7ZW45dBJuNwOZ3cLvfD2DAd5jOgIZ8baFwDw/1jhjwIQbYV+k5eNv4+bGsGp85uLUX+htyur24fD4eWv4tSQk2XB4fXr/BGEhxeEm0C//9togJQ5oXilbb/EMZC1buwOn20SUlgTKnhwUrd5A3bkCrvC8N+hiw+Ycy3tu4ly93lXLA6cEmQu+MJP675Uc+2lrEqcf2oG+XFDbsKaOTw05Kop1yp4eiChdun2F3STUJNoCDPe/iSjehTrjXC26fGxP8YPmDO+1+LK+uCXO/MSQY+F9xVcQa1eFF2jruyAFfH18D/QavgcID1TW3ncFOh83rI0Fg5/5KEhPs3LlsE3efnd0mYR/6fJZUeWo6X8193Xc3/EhGioOK6sAXYEaKo+Z+Dfo4FPpmX7+7lKIKd2BcWwxlVR6cHj/G+Hnx0yp6dk7ifyVVJCXYSE6wUVzlwR38EBjAE0rsoLojLeEfrtCP3jpdca8Br1v7500Ri0Og7ZXT7cdPcLw/yOU1uL1e7AIZKbC/wtWqPeCQ1up57yl10icjudZ9ackJ7Cl1NrfkiDToW0Bzdqi8u+FH/H4/pVWBb/bQYXYHXF4SPD5cXoNdAjs+MYYqt5dqj1Dt8Tf6EDzVOrQn37JCXY26q9UQ+Gz4ETI7J5GR4mi1HnBIU3re0WwBZHVJoczpqXVfebWXrC4prfAu9MxYy+0pdfJDWTWG2keo+PyBXoxI4L4qtxe3z4/ba3B5NeRjQX0BryHfMiL9jfsBj8/PwJ6prdoDDtlT6qw5zyCkodetbwtg8w9ltZablNOLMqcHp9uHMYYyp4f1u0trbcW0JA16i2V1SWF/pRu7TWp+yX4Ohr4/eKiaz2fw+SHRLvh1dMVSxpiakDfG1IS7BnzbSLAJmZ2TW7UHHJIVPCkvXEOvG9oCSEm0IyJkpDhqtjzCDe6TQd64AaQk2il1eslIcdTsi2sNGvRt5KX8XRFPWJqU0wuH3VYT8uFhH2IDDCYY+oakBNFfnEV0LL71JcjBM3vDCYHPQoXLy4dbCtlVXFXrqLTWEKnnXeb01Pu6jdkCGNwngwlDenPhqP5MyunFhj1l/GvtLv6y4ttDtgCaS/PiMOoL6JDQeFxTf0GD+2Qw9fg+eH2BIA8dU5xggyQ7waNpAjtODeDygsNu06GbNhbeiw+nvfiWJUBKoo2e6Uk1J9wJkGiDpATBbgN7sDfkb4Mv3Ug974Z2xDZ2C+D83P7kZKVHNdzTHLozthmi3SPf0M6ZzT+U8c3eSnqmJ1Fc6cYmggGq3T58BHZI1T0czelp/FwoqmVpwLeOwP4oP4KP4f3S2fJjBcZAssNG106JFFe66d45kdOO60mZ09PqO2MhEPZPXzYqqmUn5fSqyYRkh61mC+DCUf3qbdMWh1pG1aMXkUkiskVEtonILREe/5WIrA/+WyUiw8Me2ykiX4vIlyISV6e7RjMed7idM6HnGNQ7nSO7p/LTn3TnZ8f1xJFgO+Sog9Cmq44etI1QLz40Dg+6s7UxGrOWwpe1CYw8IoMBPdIY3i+DbqmJ2G02eqQl0b1zYs0EdW2xM7axGrsFAI3f4dsUh+3Ri4gdmAecAewG1orIMmPMprDFdgCnGmNKRGQysAA4Mezx8caYfS1WdRs53GFS0RwLe7hv6/DnyOqSQnbfDN7fvLdmszQ81A0Hd9Cq1qXDNE1jByYM6cXn35ewvyowW2Z6cgIIuDx+vH4/vvCjDQgEuzHgsAs+v6FnWhLDj+hGmdNDRbWDk49JpaTSTVbXTuwpOXhCX1vsjG2K0Ng7RHfIdVscahlNj340sM0Ys90Y4wYWA1PCFzDGrDLGhKYK/BSofzulFR1uPL0xojlMKprxuNC39Z5SZ80XQPiXQd3nWPPdPr4rrCTRLiQl2HR6gTYWaSxeAz56CXbBYbeR1a0Tx/bqTJ+MZIZmZTC4dzoI2JCaLVObDdKT7YG8l8AUzZ2T7PgJTROdQEanREYd1Y2crIxG7RRtTxq7w7cpogn6LCA8PXcH76vPlcA7YbcN8J6IrBORvPoaiUieiOSLSH5RUVEUZbWuaIZlovkFHe7LoO5z7CmtRgQ6JSbgqnvqqmpzGvLRsUswTET4trCC7D5ppCYlkJOVTo/0ZBJsQrdODuz24AEHAikJdkRsOOxCenICackJ2Gw2UhyBWAp9Ts7P7c91Pz+20UMi7UVThnsaK5qdsZH+0iMOHojIeAJBf3LY3WONMQUi0hNYISLfGGNWHvKExiwgMORDbm5uowcnWno+imiGZUK/oNAMdENSHFw4ql+t152U04sbFn/J/0qqcNhtfLilkG6pidw86bhDnmNPaTUlTjcOm1Dp9jU4J4hqOToJWfOkJQWmIi6p8tClk4Of/qQ7GSkOLugfuNpWTlY6j7z3LV06JeE3kJ7sx+sPDEH6fIaM5AQ6JSZgt9mw2/z06JwUcSdmY4dErNTY+lr7vUUT9LuB8FfuBxTUXUhEhgELgcnGmP2h+40xBcH/C0VkKYGhoEOCvjlaYz6KaMfNovkFhUIjFCi2OiEyuE8GOVkZ7K900zUlEYPB4/NT4WraRShUdHQcvvGSEgSbBM7jqPYG5tpPtEvwGgWGXp2TajpEN5xxcCrh0GckdEGb0LTa5xzfl9e+KKCkysOxvTqzvagSr9+QEaHTpJoumqBfCxwjIgOAPcBFwCXhC4jIEcCrwG+MMd+G3Z8K2Iwx5cGfJwB/aqniQ1rj8KSmHCYVyT9Wf0+FyxP4QCTYyclKx2G3H1JbaGbKrC7J7NQZJFudnvhUW0PXFAjtN02wCeeN7EdRhYuSKg+VLi/fFVXg9kG63cbgXmmceHQmZU5PvTsSr/v5sfTtklJzzYVfDMuqdSWqw80bFes9+eZozfd22DF6Y4wXmAksBzYDS4wxG0VkmohMCy52B9AdeKrOYZS9gI9F5CvgM+AtY8y7Lf0mWuPwpJYYN9v8QxkfbduPMZCaaCfFYWfd96W4vN5Daiup8pDssJGW4uCobp2odHvreVbVXNqTPyjVIaQ4bKQlB3aEhgvtNE20C4l26Nk5kXvPHUaXToFrzV4xdgCnHtuDbqmJZGWk0Dk5Ia52ksaTqE6YMsa8Dbxd5775YT9fBVwVod12YHjd+1taax2eFO24WX2PvbvhR7p2clBc6QIRHMEjaTYVlHPacT1rLTt2YKAntKekirQUByDYpeG5u1XjhB8PHxKrAR/ew3bYhB6dE+mSmkhBqZMD1d5ah9g6bA1fAaw+iTbo1jmZkip3cOzcQY/OgRP33D4/dpuQnuwgJyudkipPzQXDRx3VrdYV0Ib0PXh5ykj7qSD63mo899itFBdnxrbUMEskzfnD21PqZHCfND7cUo3PHzxszxhKnN5Dejx134PfBI5MCFxLs/bzBua+if4i1R1deA8+UtjHihSHDa/P0CnRzu1nZbN2ZzFrdxSTmZbEBcG/w/Dr/z71n21kpiXx0rSTaq4NHLp85KgBgTAOXbVs64/lFJRV11w9rJNDyEhJxO31MbBHZ2w24d6pOWzYc6DWa4T/HEmky10ezvnB6xnXfQ7VeuIi6KM5+sUKoS2No7p1Ytu+ysCHTISTB3Y/pLa67+EnmamUVrkDJ424fNiC12512Km5RqzTo4dfHk6sjcUHLiwDSXYhNdlBaaUbX/BkoT4ZyQzs2blmaASouZA21L6ucH3heEGEAF27s5hRA7oD1FyqEgNOr48URwKnHtez5ii1DXsO1OqtR6LB3P7ERdBDbB56FeqlJ9ht9OicSPfUJHqkJ/ObMUdGXD78PeRkpfPQ8m/5/H8lCD6SHDZy+mZQVB7olaUlJbC7xIn3MDkWPsd9R9IW4/AJEjjpx+uDJIeNX47M4pNt+9hdUo3HbwJj3MGzPlOT7Li9fvp2Seac4/vVTEe7JH9XrR44NG1HZN1eciT19dIb6l3rkEt8kFjr8UDgOPr8/PiYFid0pfethRUc07MzN044NuotjVDbL3aV0jMtiYcvGF5rMrS7X9/IhoIyqoKXXrNL4J/bH+ghHn9EV04flMnTH+1kf4UbEXDYAkf/lMf5oZvhf9diTCCRozQ0K73mHIrPdxazryqwYzzRJqQk2hg9oDu90pP4/H+lFJa7an43G/YcAAJf0n9Y8hUllW6SE+2kOBLI6pJMTlYG1/382Fqv1ZyrkzWnrYo/IrLOGJMb8TEN+tbXWh/m0LhsuNC4LFBrkz/8+OWTju7Op9v34zcw9ujufLilkD1lrlrP47BBv64pTB9/DEvyd7GnuApEOFDtIT3ZQZdODlKTDm4Qnjwwk9e/3MMPZS4SEwSHTeiZnozT7WVXiROf/+CUsyZ4urvPHzhfoHNSAv27pvBtYQUuryHBBl2SExjQM42unRzsLnFSWO4i0W6red195S6KK6o54Kp9tS2/sxySOmGz2cH4GZrVhT5dkukSPJU+1IN2eXwgQmKCjR8PVIMx5PTrwoDM1FpDFy/l72JJ/q6a+0O/i/B1H2moo+5UHBrGqrVp0Ldj0X5JRLNc+DLhP9/88ld8sLmQtOQEMtOS2FceCP3w8WGAm1/+CqDWfXVfN9JcQ3WDsq5Q2/Pnr2JfuYvp4wdGvZMv9Hj6/k3k5eWxa9cuJl19F86jTjmk/nA3v/wVO/ZVRhzTrk9TvrC1163aiga9ilvFxcXceOON/P3vf2fQoEEsXLiQsWPHWl2WUm2uoaDXK0ypdu22227jH//4B7fddhtffPGFhrxSEWiPXrU7e/fupbKykqOPPprCwkIKCgoYMWKE1WUpZSnt0au4YIzh+eefJzs7m6uuCpyI3bNnTw15pQ5Dg161Czt37mTixIlcccUVDBkyhPnz5x++kVIKiKMTplT8+uSTT5g4cSIiwrx585g2bRq2RhwXr1RHp58WFbM8nsBEdSNHjuSSSy5hw4YNTJ8+XUNeqUbST4yKOR6Ph3vvvZfhw4dTWVlJSkoKCxYs4MgjI08doZRqmAa9iimff/45o0aN4vbbb2fo0KG4XK7DN1JKNUiDXsUEt9vNLbfcwujRoyksLGTp0qX861//olu3+qfIVUpFR4NexYSEhARWrVrF5ZdfzqZNmzjnnHOsLkmpuBFV0IvIJBHZIiLbROSWCI//SkTWB/+tEpHh0bZVHVd5eTl/+MMf2Lt3LzabjRUrVrBw4UK6dOlidWlKxZXDBr2I2IF5wGQgG7hYRLLrLLYDONUYMwyYCyxoRFvVAb3zzjsMGTKEv/zlL7z33nsAJCUlWVyVUvEpmh79aGCbMWa7McYNLAamhC9gjFlljCkJ3vwU6BdtW9Wx7N+/n0svvZQzzzyTzp0788knn3DppZdaXZZScS2aoM8Cwuee3R28rz5XAu80tq2I5IlIvojkFxUVRVGWao9uu+02Fi1axJw5c/jiiy8YM2aM1SUpFfeiOTM20vXXIs6EJiLjCQT9yY1ta4xZQHDIJzc3N/ZmWlNNVlBQQFVVFQMHDmTu3LlcffXVDB8eeZ54pVTLi6ZHvxsIv2pCP6Cg7kIiMgxYCEwxxuxvTFsVn4wxPPPMM2RnZ/O73/0OgB49emjIK9XGogn6tcAxIjJARBKBi4Bl4QuIyBHAq8BvjDHfNqatik/bt2/njDPO4KqrrmLEiBE8/fTTVpekVId12KEbY4xXRGYCywE78KwxZqOITAs+Ph+4A+gOPCUiAF5jTG59bVvpvagY8fHHHzNx4kTsdjvz58/nd7/7nc5Po5SF9MIjqsV4PB4cDgdOp5Mbb7yRW2+9lf799VqpSrUFvfCIalVut5u5c+cybNiwmknI/vrXv2rIKxUjNOhVs6xdu5bc3FzuuOMORowYgdvttrokpVQdGvSqSdxuNzfffDM//elP2b9/P6+//jqLFi2ia9euVpemlKpDg141SUJCAmvWrOHKK69k06ZNnH322VaXpJSqhwa9ilpZWRnXX389P/zwAzabjffee48FCxaQkZFhdWlKqQZo0KuovPXWWwwZMoQnnniC999/H9BJyJRqLzToVYOKior41a9+xVlnnUXXrl1ZvXo1v/nNb6wuSynVCBr0qkFz5szhpZde4q677mLdunWMHj3a6pKUUo2kJ0ypQ+zZs4eqqiqOOeYYioqK+PHHH8nJybG6LKVUA/SEKRUVYwxPP/002dnZ5OXlAYFJyDTklWrfNOgVAN999x2nn346eXl5nHDCCSxcuNDqkpRSLSSa+ehVnPvoo4+YOHEiDoeDBQsWcNVVVxGcnE4pFQe0R9+BhaYryM3N5YorrmDTpk387ne/05BXKs5o0HdALpeLu+66i6FDh1JRUUFKSgrz5s0jK6uhK0QqpdorDfoOZs2aNZxwwgncfffdjB49Gq/Xa3VJSqlWpkHfQbjdbm688UbGjBlDWVkZb775Ji+++CJdunSxujSlVCvToO8gEhIS+OKLL5g2bRobN27kF7/4hdUlKaXaSFRBLyKTRGSLiGwTkVsiPD5IRFaLiEtEZtV5bKeIfC0iX4qIngXVhkpLS7n22mspKCjAZrOxfPlynnrqKdLT060uTSnVhg4b9CJiB+YBk4Fs4GIRya6zWDFwLfBQPU8z3hgzor6ztlTLW7ZsGUOGDGHevHl88MEHACQmJlpclVLKCtH06EcD24wx240xbmAxMCV8AWNMoTFmLeBphRpVIxQWFnLRRRcxZcoUunfvzpo1a/j1r39tdVlKKQtFE/RZwK6w27uD90XLAO+JyDoRyWtMcarx7rjjDpYuXcrcuXPJz88nN1c3opTq6KI5MzbS2TONmQltrDGmQER6AitE5BtjzMpDXiTwJZAHcMQRRzTi6dWuXbtwOp0ce+yx3HPPPVx77bVkZ9cdXVNKdVTR9Oh3A/3DbvcDCqJ9AWNMQfD/QmApgaGgSMstMMbkGmNye/ToEe3Td2h+v5+//vWvDBkyhGnTpgGQmZmpIa+UqiWaoF8LHCMiA0QkEbgIWBbNk4tIqoikhX4GJgAbmlqsOmjr1q2MHz+e6dOnc+KJJ/LMM89YXZJSKkYddujGGOMVkZnAcsAOPGuM2Sgi04KPzxeR3kA+kA74ReR6AkfoZAJLg3OnJAD/NMa82yrvpAP56KOPmDBhAsnJyTz77LNcfvnlOj+NUqpeeuGRdsTlcpGUlER1dTU333wzf/zjH+nTp4/VZSmlYoBeeKSdc7lczJkzp2YSsuTkZB5//HENeaVUVDToY9zq1as5/vjjueeeexgzZoxOQqaUajQN+hjlcrm4/vrrGTt2LBUVFbz99tv8/e9/10nIlFKNpkEfoxwOB+vXr2f69Ols3LiRyZMnW12SUqqd0qCPISUlJcyYMaPWJGRPPvkkaWlpVpemlGrHNOhjxNKlS8nOzuZvf/sbH374IRDo1SulVHNp0Fts7969nH/++Zx77rn07t2bzz77jEsuucTqspRScUSD3mJ33303b7zxBv/3f//HZ599xsiRI60uSSkVZ/SEKQt8//33VFdXc9xxx7F//36KiooYNGiQ1WUppdoxPWEqRvj9fubNm0dOTk7NJGTdu3fXkFdKtSoN+jayZcsWTj31VGbOnMlJJ53Ec889Z3VJSqkOIpr56FUz/fe//2XixIl06tSJ559/nksvvVQnIVNKtRnt0bei6upqAE488USmTZvGpk2buOyyyzTklVJtSoO+FVRXV3PrrbeSk5NDeXk5ycnJPProo/Tu3dvq0pRSHZAGfQv7+OOPGTFiBH/+858ZN24cfr/f6pKUUh2cBn0LcblcXHPNNYwbN47q6mqWL1/Os88+S0ZGhtWlKaU6OA36FuJwONi0aRPXXHMNGzZsYMKECVaXpJRSgAZ9sxQXF3P11VezZ8+emknIHnvsMTp37mx1aUopVSOqoBeRSSKyRUS2icgtER4fJCKrRcQlIrMa07a9euWVV8jOzmbhwoWsXLkSgIQEPVpVKRV7Dhv0ImIH5gGTCVzw+2IRya6zWDFwLfBQE9q2Kz/88AO//OUvOe+888jKymLt2rVcfPHFVpellFL1iqZHPxrYZozZboxxA4uBKeELGGMKjTFrAU9j27Y3f/rTn3jrrbe47777WLNmDSNGjLC6JKWUalA0QZ8F7Aq7vTt4XzSibisieSKSLyL5RUVFUT5929ixYwfffPMNAPfccw9fffUVs2fP1qEapVS7EE3QRzqNM9opL6Nua4xZYIzJNcbk9ujRI8qnb10+n4/HH3+cnJwcpk+fDgQmITvuuOMsrkwppaIXTdDvBvqH3e4HFET5/M1pa6nNmzdzyimncN1113Hqqafy/PPPW12SUko1STRBvxY4RkQGiEgicBGwLMrnb05by/z3v/9lxIgRbNmyhRdffJG33nqLI444wuqylFKqSQ47yGyM8YrITGA5YAeeNcZsFJFpwcfni0hvIB9IB/wicj2QbYw5EKltK72XZnM6naSkpPDTn/6UmTNnMnv2bHr27Gl1WUop1Sx6hSkCAX/33Xfz0ksv8eWXX5KWltZmr62UUi1BrzDVgJUrVzJ8+HDuv/9+xo8fr5OQKaXiTocNepfLxYwZMzj11FPxer28//77LFy4UCchU0rFnQ4b9ImJiWzZsoXrr7+er7/+mtNPP93qkpRSqlV0qKDft28feXl57N69GxHh3Xff5S9/+QupqalWl6aUUq2mQwS9MYYlS5aQnZ3Nc889x8cffwzoJGRKqY4h7oO+oKCAqVOncuGFF3LkkUfy+eefc9FFF1ldllJKtZm4D/q5c+eyfPlyHnroIVavXs3QoUOtLkkppdpUXB5Hv337dlwuF4MHD6a4uJji4mIGDhzYghUqpVRs6TDH0ft8Ph599FGGDh3KjBkzAOjWrZuGvFKqQ4uboN+4cSNjx47lhhtuYPz48bzwwgtWl6SUUjEhLg47+fDDD5kwYQIZGRn885//5KKLLkIk0gzJSinV8bTrHn1VVRUAY8aM4brrrmPTpk1cfPHFGvJKKRWmXQZ9VVUVs2bNYsiQIZSXl5OUlMSDDz5IrFywRCmlYkm7C/oPP/yQYcOG8fDDDzNx4kSry1FKqZjXboK+urqa3//+94wfPx6ADz74gPnz5+uUwkopdRjtJuiTkpLYsWMHs2bNYv369TWBr5RSqmExHfRFRUVcddVVNZOQvfPOOzz44IN06tTJ6tKUUqrdiNmgX7RoEdnZ2bzwwgusWrUKALvdbnFVSinV/sTkFAhdunQxZWVljB49mmeeeYacnByrS1JKqZjW7qZAOHDgAI888girVq3SkFdKqWaKyR69iBQB31tdRwMygX1WF9EAra/5Yr1Gra/5Yr3GxtZ3pDEm4slEMRn0sU5E8uvbRIoFWl/zxXqNWl/zxXqNLVlfTA7dKKWUajka9EopFec06JtmgdUFHIbW13yxXqPW13yxXmOL1adj9EopFee0R6+UUnFOg14ppeKcBn0YEZkkIltEZJuI3BLh8UEislpEXCIyqzFtY6C+nSLytYh8KSJNv/J682v8lYisD/5bJSLDo20bA/W1+jqMor4pwdq+FJF8ETk52rYxUqPl6zBsuVEi4hOR8xrb1sL6mrb+jDH6L7Cfwg58B/wESAS+ArLrLNMTGAXcC8xqTFsr6ws+thPIjIF1eBLQNfjzZGBNjK3DiPW1xTqMsr7OHNy3Ngz4pq3WX3NrjJV1GLbcB8DbwHmx9DdYX33NWX/aoz9oNLDNGLPdGOMGFgNTwhcwxhQaY9YCnsa2tbi+thJNjauMMSXBm58C/aJta3F9bSGa+ipM8BMPpAIm2rYxUGNbiHY9XAO8AhQ2oa1V9TWZBv1BWcCusNu7g/e1dttoNfc1DPCeiKwTkbwWreygxtZ4JfBOE9s2RXPqg9Zfh1HVJyJTReQb4C3gt41pa3GNEAPrUESygKnA/Ma2tbg+aOL6S2hCofEq0hXFo+2JNKdttJr7GmONMQUi0hNYISLfGGNWtlBtIVHXKCLjCQRpaPw2ptZhhPqg9ddhVPUZY5YCS0VkHDAX+Hm0bVtAc2qE2FiHjwKzjTE+kVqLx8rf4KNErg+auP406A/aDfQPu90PKGiDttFq1msYYwqC/xeKyFICm5AtHfRR1Sgiw4CFwGRjzP7GtLWwvrZYh41aB8aYlSJytIhkNratFTUaY/bFyDrMBRYHQzQTOFNEvFG2taw+Y8xrTV5/rbVTpL39I/Cltx0YwMGdJEPqWfYuau+MjbqtRfWlAmlhP68CJlmxDoEjgG3ASU19fxbV1+rrMMr6BnJwR+dIYA+BXmKrr78WqDEm1mGd5Z/n4M7YmPgbbKC+Jq8/7dEHGWO8IjITWE5gj/ezxpiNIjIt+Ph8EekN5APpgF9Eriewx/xApLaxUh+BXsHSYA8hAfinMebdlqwv2hqBO4DuwFPBerzGmNz62sZKfUAvWnkdRlnfL4FLRcQDOIELTeCT3+rrr7k1ikisrMNGtY2V+mjG36BOgaCUUnFOj7pRSqk4p0GvlFJxToNeKaXinAa9UkrFOQ16pZSKcxr0SikV5zTolVIqzv1/qZretRbHfEsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.errorbar(search_data['bandgap'], y_pred, fmt='o', yerr=y_std, alpha=0.5)\n",
    "\n",
    "ax.set_xlim(ax.get_xlim())\n",
    "ax.set_ylim(ax.get_xlim())\n",
    "\n",
    "ax.plot(ax.get_xlim(), ax.get_xlim(), 'k--')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sharp-pantyhose",
   "metadata": {},
   "source": [
    "It is not a great model, but it at least gives us a starting place that is a little better than random"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "level-society",
   "metadata": {},
   "source": [
    "### Create the Oracle\n",
    "The \"oracle\" in active learning is the tool used to gather properties for a certain data point. \n",
    "We are going to make one that is just a lookup table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "greek-worst",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:24.305264Z",
     "iopub.status.busy": "2021-02-25T12:47:24.304754Z",
     "iopub.status.idle": "2021-02-25T12:47:24.307023Z",
     "shell.execute_reply": "2021-02-25T12:47:24.307403Z"
    }
   },
   "outputs": [],
   "source": [
    "lookup_table = dict(zip(data['smiles_0'], data['bandgap']))  # Makes a dictionary with key as a SMILES and value as a bandgap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "neutral-stock",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:24.311214Z",
     "iopub.status.busy": "2021-02-25T12:47:24.310725Z",
     "iopub.status.idle": "2021-02-25T12:47:24.313559Z",
     "shell.execute_reply": "2021-02-25T12:47:24.313084Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_bandgap(smiles: str):\n",
    "    \"\"\"Lookup the bandgap of a model from a table\n",
    "    \n",
    "    Args:\n",
    "        smiles: SMILES string of a molecule\n",
    "    Returns:\n",
    "        (float) A bandgap energy\n",
    "    \"\"\"\n",
    "    return lookup_table[smiles]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "single-wesley",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:24.317130Z",
     "iopub.status.busy": "2021-02-25T12:47:24.316668Z",
     "iopub.status.idle": "2021-02-25T12:47:24.320420Z",
     "shell.execute_reply": "2021-02-25T12:47:24.320853Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.44260000000000005"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_bandgap('CC')  # Get the bandgap energy of ethane"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "global-excellence",
   "metadata": {},
   "source": [
    "## Run the Active Learning Agent\n",
    "The `CommitteeRegressor` we created also gives us the tools we need to run active learning. We can definte a query strategy, such as [UCB](https://modal-python.readthedocs.io/en/latest/content/query_strategies/Acquisition-functions.html#upper-confidence-bound), to pick the points and then use the regressor to determine which new point would be the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "elementary-comfort",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:24.324187Z",
     "iopub.status.busy": "2021-02-25T12:47:24.323708Z",
     "iopub.status.idle": "2021-02-25T12:47:24.326566Z",
     "shell.execute_reply": "2021-02-25T12:47:24.326093Z"
    }
   },
   "outputs": [],
   "source": [
    "committee.query_strategy = max_UCB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sound-concentrate",
   "metadata": {},
   "source": [
    "Given a list of points, it will suggest the next best point to evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aboriginal-building",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:24.350173Z",
     "iopub.status.busy": "2021-02-25T12:47:24.345222Z",
     "iopub.status.idle": "2021-02-25T12:47:31.830067Z",
     "shell.execute_reply": "2021-02-25T12:47:31.829297Z"
    }
   },
   "outputs": [],
   "source": [
    "point, smiles = committee.query(search_data['smiles_0'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "little-miami",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:31.833938Z",
     "iopub.status.busy": "2021-02-25T12:47:31.833353Z",
     "iopub.status.idle": "2021-02-25T12:47:31.839083Z",
     "shell.execute_reply": "2021-02-25T12:47:31.837177Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected OCC12CC3OC(C1)C23, which is point #984 in our dataset\n"
     ]
    }
   ],
   "source": [
    "print(f'Selected {smiles[0]}, which is point #{point[0]} in our dataset')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sitting-steering",
   "metadata": {},
   "source": [
    "We can use our Oracle to get its band gap energy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "scheduled-saskatchewan",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:31.843765Z",
     "iopub.status.busy": "2021-02-25T12:47:31.843273Z",
     "iopub.status.idle": "2021-02-25T12:47:31.846610Z",
     "shell.execute_reply": "2021-02-25T12:47:31.846217Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OCC12CC3OC(C1)C23 has a bandgap of 0.30\n"
     ]
    }
   ],
   "source": [
    "new_bandgap = get_bandgap(smiles[0])\n",
    "print(f'{smiles[0]} has a bandgap of {new_bandgap:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lasting-breeding",
   "metadata": {},
   "source": [
    "We can then add this to our training set and repeat the process. We will do that in a for loop to do it many times.\n",
    "\n",
    "But first, let's move the data into lists for making our coding easier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "exempt-wallace",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:31.851163Z",
     "iopub.status.busy": "2021-02-25T12:47:31.849176Z",
     "iopub.status.idle": "2021-02-25T12:47:31.854428Z",
     "shell.execute_reply": "2021-02-25T12:47:31.853944Z"
    }
   },
   "outputs": [],
   "source": [
    "train_X = seed_data['smiles_0'].tolist()\n",
    "train_y = seed_data['bandgap'].tolist()\n",
    "search_X = search_data['smiles_0'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "circular-apartment",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:47:31.860475Z",
     "iopub.status.busy": "2021-02-25T12:47:31.859884Z",
     "iopub.status.idle": "2021-02-25T12:53:55.237427Z",
     "shell.execute_reply": "2021-02-25T12:53:55.236990Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [06:23<00:00,  7.67s/it]\n"
     ]
    }
   ],
   "source": [
    "for step in tqdm(range(50)):\n",
    "    # Run the query with the AI tool\n",
    "    points, smiles = committee.query(search_X)\n",
    "    \n",
    "    # Get the value of the bandgap\n",
    "    bandgaps = [get_bandgap(s) for s in smiles]\n",
    "    \n",
    "    # Remove that value from the search space\n",
    "    search_X = np.delete(search_X, points)\n",
    "    \n",
    "    # Add it to the training set\n",
    "    train_X.extend(smiles)\n",
    "    train_y.extend(bandgaps)\n",
    "    \n",
    "    # Retrain the ensemble\n",
    "    model.fit(np.expand_dims(train_X, axis=-1), train_y)\n",
    "    \n",
    "    # Reset the active learning agent\n",
    "    committee = CommitteeRegressor(\n",
    "        model.estimators_,\n",
    "        query_strategy=max_UCB,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brazilian-wildlife",
   "metadata": {},
   "source": [
    "Plot it compared to random selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bearing-corpus",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:53:55.245052Z",
     "iopub.status.busy": "2021-02-25T12:53:55.244461Z",
     "iopub.status.idle": "2021-02-25T12:53:55.258693Z",
     "shell.execute_reply": "2021-02-25T12:53:55.258262Z"
    }
   },
   "outputs": [],
   "source": [
    "# Simulate a random search by drawing a random subset\n",
    "random_selection = search_data.sample(len(train_X) - len(seed_data))\n",
    "random_y = np.concatenate([seed_data['bandgap'].values, random_selection['bandgap'].values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "expensive-mistress",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-25T12:53:55.288983Z",
     "iopub.status.busy": "2021-02-25T12:53:55.281683Z",
     "iopub.status.idle": "2021-02-25T12:53:55.470956Z",
     "shell.execute_reply": "2021-02-25T12:53:55.470407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe51b415d50>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAos0lEQVR4nO3deXhV1fX/8fdKGMIkU9ACEcEiKipjFFFUqIpiHcpXFLSK2Cpaay2tVoutVTtha1uR1kr5WYY6MIiiqFipgqUIFoKCBEVBBYmAJGFKgECG9fvj3tBLOEluQi43N/m8nocn95y998naglk5Z++zt7k7IiIiZSXFOwAREamdlCBERCSQEoSIiARSghARkUBKECIiEqhBvAOoSampqd65c+d4hyEikjBWrFiR4+7tgsrqVILo3LkzGRkZ8Q5DRCRhmNnG8sr0iElERAIpQYiISKCYJggzu9TMPjaz9Wb20wrqnWlmxWY2LOLcBjNbbWYrzUzPjUREjrKYjUGYWTLwBHAxkAUsN7O57v5hQL3fAW8EXGaQu+fEKkYRESlfLO8gzgLWu/tn7n4AmAFcFVDvB8ALwLYYxiIiIlUUywTREdgUcZwVPneQmXUEhgITA9o7MN/MVpjZ6PK+iZmNNrMMM8vIzs6ugbBFRARimyAs4FzZpWPHA/e5e3FA3XPdvQ8wBPi+mZ0f9E3cfZK7p7t7ert2gVN5RUSkGmL5HkQWcHzEcRqwuUyddGCGmQGkApeZWZG7v+TumwHcfZuZzSH0yGpRRd/w448/ZuDAgYecu/baa7njjjvYu3cvl1122WFtRo0axahRo8jJyWHYsGGHlX/ve99j+PDhbNq0iRtvvPGw8rvvvpsrrriCjz/+mNtuu+2w8p///OdcdNFFrFy5kjFjxhxW/tvf/pZzzjmHJUuWcP/99x9WPn78eHr16sWbb77Jr3/968PK//a3v3HyySfzyiuv8Mc//vGw8qeffprjjz+emTNn8uSTTx5WPnv2bFJTU5k6dSpTp049rHzevHk0bdqUv/71r8yaNeuw8rfffhuAP/zhD7z66quHlDVp0oTXX38dgF/96le89dZbh5S3bduWF154AYCxY8eydOnSQ8rT0tJ45plnABgzZgwrV648pLxbt25MmjQJgNGjR/PJJ58cUt6rVy/Gjx8PwA033EBWVtYh5f3792fcuHEAXH311eTm5h5SfuGFF/LAAw8AMGTIEPbt23dI+eWXX84999wDcNi/O9C/Pf3bGw/U/n97FYnlHcRy4CQz62JmjYARwNzICu7exd07u3tnYDZwh7u/ZGbNzKwFgJk1AwYDmTGMVUREyrBYbhhkZpcReoyUDEx299+Y2e0A7j6xTN2pwKvuPtvMTgTmhIsaAM+5+28q+37p6emuN6lFRKJnZivcPT2wrC7tKKcEISJSNRUlCL1JLSIigZQgREQkkBKEiIgEUoIQEZFAShAiIhJICUJERAIpQYiISCAlCBERCaQEISIigZQgREQkkBKEiIgEUoIQEZFAShAiIhJICUJERAIpQYiISCAlCBERCaQEISIigZQgREQkkBKEiIgEUoIQEZFAShAiIhJICUJERAIpQYiISCAlCBERCaQEISIigZQgREQkkBKEiIgEUoIQEZFAShAiIhJICUJERAIpQYiISCAlCBERCRTTBGFml5rZx2a23sx+WkG9M82s2MyGVbWtiIjERswShJklA08AQ4DuwHVm1r2cer8D3qhqWxERiZ1Y3kGcBax398/c/QAwA7gqoN4PgBeAbdVoKyIiMRLLBNER2BRxnBU+d5CZdQSGAhOr2jbiGqPNLMPMMrKzs484aBERCYllgrCAc17meDxwn7sXV6Nt6KT7JHdPd/f0du3aVT1KEREJ1CCG184Cjo84TgM2l6mTDswwM4BU4DIzK4qyrYiIxFAsE8Ry4CQz6wJ8CYwAro+s4O5dSj+b2VTgVXd/ycwaVNZWRERiK2YJwt2LzOxOQrOTkoHJ7r7GzG4Pl5cdd6i0baxiFRGRw5l74KP9hJSenu4ZGRnxDkNEJGGY2Qp3Tw8q05vUIiISSAlCREQCKUGIiEggJQgREQmkBCEiIoGUIEREJJAShIiIBFKCEBGRQEoQIiISSAlCREQCKUGIiEggJQgREQmkBCEiIoGUIEREJJAShIiIBFKCEBGRQEoQIiISSAlCREQCKUGIiEggJQgREQmkBCEiIoGUIEREJFCDeAcgIlLX5Obv588L1pO/v+iofL/mjRvw0JWn1fh1K00QZpYC3AEMABxYDDzp7gU1Ho2ISB3w4Nw1/DNzK8cdk3JUvl+bZo1ict1o7iD+AeQBfw4fXwc8DVwTk4hERBLYvz/J5tUPtvDji7tx14UnxTucIxJNgjjZ3XtGHC80s1WxCkhEJFEVFBbzwEuZnJjajNsuODHe4RyxaAap3zezs0sPzKwf8E7sQhIRSUxPLFzPF9v38utvnU7jBsnxDueIRXMH0Q8YaWZfhI87AR+Z2WrA3b1HzKITEUkQ67flM/HfnzK0d0fO6Zoa73BqRDQJ4tKYRyEiksDcnZ/NWU2Thsn87JunxjucGlNpgnD3jQBmdiyQEnH+i3IbiYjUIy++9yX//Xw7vx16BqnNG8c7nBpT6RiEmV1pZuuAz4F/AxuA12Mcl4hIQtix5wC/mfcRfTq1YsSZx8c7nBoVzSOmXwFnA2+6e28zG0RoqmulzOxS4HEgGXjK3R8pU35V+PolQBEwxt0Xh8s2EJpeWwwUuXt6VD0SETlCSz7N4TtTl3OgqKTSug4kmfGboWeQlGSxD+4oiiZBFLp7rpklmVmSuy80s99V1sjMkoEngIuBLGC5mc119w8jqr0FzHV3N7MewCzglIjyQe6eE313RESOTEmJM27eWto0bcTVfdOiatP3hNac2v6YGEd29EWTIHaaWXNgEfCsmW0j9Nt+Zc4C1rv7ZwBmNgO4CjiYINw9P6J+M0LJWEQkbl7P3MrqL3fxx2t6Rp0g6qpo3oO4CtgH/Aj4J/ApcEUU7ToCmyKOs8LnDmFmQ81sLfAa8J2IIgfmm9kKMxtd3jcxs9FmlmFmGdnZ2VGEJSISrKi4hD/O/5huxzXnW70P+3FV70Qzi2lPxOG0Klw76GHcYXcI7j4HmGNm5xMaj7goXHSuu28Oz576l5mtdfdFAe0nAZMA0tPTdQciItU2e0UWn+XsYdKNfUmuY+MJ1RHNLKY8M9td5s8mM5tjZhW9S54FRA7ppwGby6sc/uH/dTNLDR9vDn/dBswh9MhKRCQmCgqLGf/mOnp3asXF3Y+Ldzi1QjSPmP4E/ITQ46E04B7g/wEzgMkVtFsOnGRmXcysETACmBtZwcy6mpmFP/cBGgG5ZtbMzFqEzzcDBgOZVemYiEhVPL10I1t3F3DvJacQ/rFU70X1JrW794s4nmRm77r7L83s/vIauXuRmd0JvEFomutkd19jZreHyycCVxNaxqOQ0DjH8PCMpuMIPXYqjfE5d/9ntXooIlKJ3QWFPPH2es7v1o7+X28b73BqjWgSRImZXQvMDh8Piyir8Jm/u88D5pU5NzHi8++Aw6bMhmc+9Sx7XkQkFp5a9Bk79xZy7yUnxzuUWiWaR0zfBm4EtgFfhT/fYGZNgDtjGJuISMxl5+3nqcWf880e7Tm9Y8t4h1OrRDOL6TPKn9a6uGbDEamanXsPcM/zq1iVtSveoUiCKigsZn9RCXdf3C3eodQ65SYIM/szFTxCcve7YhKRSJQ25u7h5inLydqxj6t6daBBcjQ3xCKH692pFSe2ax7vMGqdiu4gMsJfzwW6AzPDx9cAK2IZlEhlVmzczq3/WEGJO8/e2o8zO7eJd0gidU65CcLdpwGY2ShCayIVho8nAvOPSnQiAV5ZtZm7n19Fh5YpTLn5LLqkNot3SCJ1UjSzmDoALYDt4ePm4XMiMZNXUMiyz7dTUuYh5wdZO/nzgvWc2bk1f7sxnTbNGsUnQJF6IJoE8QihfakXho8vAB6KWURS72V+uYvvPbuCTdv3BZZf2bMDvx/Wg5SGib/nr0htFs0spilm9jqhvakBfuruW2MbltRXM5d/wQMvr6Fts0ZMHpXOsS1SDilv3CCJrsc215uuIkdBNHcQEHoTOjtcv5uZdQtaOE+kugoKi3ngpUyeX5HFgK6pPD6iF23r0NaNIomo0gQR3hxoOLCG0M5vEJr+qgQhNWJj7h5uf+Y9Ptqym7u+0ZUfXtRNK2mK1ALR3EF8CzjZ3ffHOBapp279RwZf7d7PlFFnMuiUY+MdTuwc2Auzb4Y92rdEaliT1nDDCzV+2WgSxGdAQ0AJQmrcpu17+eSrfH5xefe6nRwAPpoLn/wTOp8HDfT4TGpQSquYXDaaBLEXWGlmbxGRJPQmtdSEJZ+Gthw/76TUOEdyFLz/DLTuDDe9AhpklwQQTYKYS5l9HERqyuL1uRzbojFdj63jyxzs2AAb/gODfqbkIAkjmmmuVdlmVCRqJSXOkvU5XNCtXd2ftrpyOmDQ87p4RyIStWhmMZ0EjCO0HtPBSenuXtF2oyKVWrs1j9w9Bzi3ax1/vFRSAiufgy7nQ6vjK68vUktEs/zlFOBJoAgYBPwDeDqWQUn98M760PhDnU8QGxfDri+g9w3xjkSkSqJJEE3c/S3A3H2juz8EfCO2YUl9sHh9Dl2Pbc7XWqZUXjmRvf8sND4GTrk83pGIVEk0CaLAzJKAdWZ2p5kNBer4fESJtf1FxSz7fDsD6vrdQ8Fu+PBlOP3/oFHTeEcjUiXRJIgxQFPgLqAvoS1Hb4phTFIPvP/FTvYVFtf9x0tr5kDRPuilx0uSeKKZxbQ8/DEfuDm24Uh98c76HJKTjH4n1vGNflY+C21PgrT0eEciUmXl3kGYWaqZPWhmd5lZczN70swyzexlM+t6NIOUumfx+hx6prXkmJSG8Q4ldnLWw6b/Qu9v690HSUgVPWJ6DmgMnAQsI7TkxjDgVeCp2IcmddXugkJWbdpZ98cfVj4LlgQ9RsQ7EpFqqegR03Hufr+F3mDa6O6Phs+vNbPvH4XYpI5699NcSryOT28tKYZVM6DrRXBM+3hHI1ItFSWIYgB3dzPLKVNWElBf4sEdFj8GW1bFO5Koddi8mycb7SV9+SzIiHc0MbI/D/I2w6Xj4h2JSLVVlCBONLO5gEV8JnzcJeaRSXQWPwZvPQytu0ByYuzP3GL3Hno0TiI5e0e8Q4mtEwfByUPiHYVItVWUIK6K+PyHMmVljyUe1r4Gb/0STr8arv57QgyEbtm1jwvGLeDn3zyVW87Tai0itVm5CcLd/300A5Eq2poJL9wKHXrBVU8kRHIAeGd9LlDHxx9E6ohoXpST2mZPDky/DlKOgRHToWGTeEcUtXfW55DavBEnH9ci3qGISCWi2Q9CapOiAzDzRtizDW6eVyMzZPL3F7F3f1ENBFcxJ/T+wzlfTyVJe06L1HrRLPed4u4FZc6lunvZmU0Shc8/XE5hwZ5qt2/70TO0/WIJGwZOYGfJ12HTziq1P1BUwuc5+az7Kp9PtuWz7qs8tuwqqLxhDRpQH3aPE6kDormDWG5mt7r7uwBmdjWh/SG6VdbQzC4FHgeSgafc/ZEy5VcBvyI0bbYIGOPui6Npm4gy//Myp7818oiv8+eib/HHf6YC71T7Go0bJNH12OacfWJbuh7bnFZNj84bzY0bJHNFT70XIJIIokkQ1wOTzextoAPQliiW+zazZOAJ4GIgi1CimevuH0ZUewuYG37XogcwCzglyrYJZ8/7s9njKay/4HGqO/xT3LA5p6f2ZUo1B6WTkozObZuS1ropyXrMIyIViGaxvtVm9htCmwTlAee7e1YU1z4LWO/unwGY2QxCU2cP/pB39/yI+s0IPaaOqm2iKSku5uvbF7G2xdn0/YaWXhCR2q/SX2PN7O+ElvzuQWg111eiXGqjI7Ap4jgrfK7s9Yea2VrgNeA7VWkbbj/azDLMLCM7OzuKsOLjk/cWkspO/JRvxjsUEZGoRPOcIxMY5O6fu/sbwNlAnyjaBT2/8MNOuM9x91OAbxEaj4i6bbj9JHdPd/f0du3aRRFWfOxc8SIHPJluA66OdygiIlGpNEG4+2Pu7hHHu9z9u1FcOwuI3KE9DdhcwfdZBHzdzFKr2ra285IS0r5awNomvTimVdt4hyMiEpVoHjGdZGazzexDM/us9E8U114OnGRmXcysETACmBtZwcy6hleLxcz6AI2A3GjaJpIvPn6fNN/CvhO1Lo+IJI5oZjFNAR4EHgMGERqHqHT6i7sXmdmdwBuEpqpOdvc1ZnZ7uHwicDUw0swKgX3A8PDdSmDbKveultj83+c5AThxwDXxDkVEJGoW8fQouILZCnfva2ar3f2M8Ln/uPt5RyXCKkhPT/eMjNq3fvS6X6dTQhIn/3xZvEMRETlE+Gd84J640QxSF5hZErDOzO40s6HAsTUaYR22ddN6Tipax/ZOl8Q7FBGRKokmQYwBmgJ3AX2BG4GbYhhTnbJxyWwAOp6t2UsikliieVFuefhjPqHxB6mCZp+9zsakNE7o1iveoYiIVEm5CSJiB7lA7n5lzYdTt+zans0pBR+wvOONnBDvYEREqqiiO4j+hN5mng78lyhmLsmh1i2eTbqV0Db9/+IdiohIlVWUIL5GaLG86wgt2PcaMD2Rp5sebUkfv8Y22tC1Z62b8CUiUqlyB6ndvdjd/+nuNxFaXmM98LaZ/eCoRZfACvbmc0r+Mj5PHUhScnK8wxERqbIKB6nNrDHwTUJ3EZ2BCcCLsQ8r8a1d8gq9bD9Ne2ioRkQSU0WD1NOA04HXgYfdPfOoRZXgiouKaLjsSXbTjJP7aXkNEUlMFd1B3AjsIbRz3F32vw1qDHB3PybGsSWsZf+4n/4HVrHsjIc5q3FKvMMREamWchOEu1dvy7N6LnPxXPptnMTyVoM5c+hd8Q5HRKTalARqUM7WL/jamz9gU3Iap936FJak/7wikrj0E6yGFBcV8dXkG2jmeykZNoWmzVvGOyQRkSOiBFFDlk37KacdWMXqXr+gS/cz4x2OiMgRi2Y/iHrJS0rYnh3dJnabMhfT74unWN7qUs4aqtdERKRuUIIox7tPjaH/5mlR1W0LbEg+ntNunRTboEREjiIliHI02fExW2nHxlNvrbyyJfH184dr3EFE6hQliHI0K8xlW0pn+g2/L96hiIjEhQapy9GiaAf7G7eNdxgiInGjBBHAS0po4zspaqqdVUWk/lKCCLB7Zy6NrAhr3i7eoYiIxI0SRICd2V8C0KDl1+IciYhI/ChBBMjPCSWIlFZKECJSfylBBCjYuQWA5m07xDkSEZH4UYIIULhrKwCt2qXFORIRkfhRggjg+dso9GSOaa1BahGpv5QgAiTvzWaHtdRe0iJSrylBBGi8P4fdya3jHYaISFwpQQRodiCXPY30FrWI1G9KEAGOKd7B/sap8Q5DRCSulCDKKCkuprXvoripBqhFpH7Taq5l7N6RTSsrxpprHSaRSIWFhWRlZVFQUBDvUKQaUlJSSEtLo2HDhlG3iWmCMLNLgceBZOApd3+kTPm3gdL1tPOB77n7qnDZBiAPKAaK3D09lrGW2pmdRSugoZbZEDlEVlYWLVq0oHPnzphZvMORKnB3cnNzycrKokuXLlG3i1mCMLNk4AngYiALWG5mc939w4hqnwMXuPsOMxsCTAL6RZQPcvecWMUYJD83tM1oSqv2R/PbitR6BQUFSg4Jysxo27Yt2dnZVWoXyzGIs4D17v6Zux8AZgBXRVZw9yXuviN8+C4Q91eXC3aULrOhBCFSlpJD4qrO310sE0RHYFPEcVb4XHm+C7wecezAfDNbYWajy2tkZqPNLMPMMqqaHYMU7f4KgFbHHn/E1xIRSWSxTBBB6coDK5oNIpQgIvf3PNfd+wBDgO+b2flBbd19krunu3t6u3ZHPvPI87/igDfgmFZ6D0KkNpozZw5mxtq1ayutO378ePbu3Xvw+LLLLmPnzp1HHEPnzp3JyTl6T79rKu6qimWCyAIifw1PAzaXrWRmPYCngKvcPbf0vLtvDn/dBswh9Mgq5hrszWG7tcKSNANYpDaaPn06AwYMYMaMGZXWLZsg5s2bR6tWrWIYXfUUFRVVWB6vuGM5i2k5cJKZdQG+BEYA10dWMLNOwIvAje7+ScT5ZkCSu+eFPw8GfhnDWA9qvD+XvAat0RwmkfI9/MoaPty8u0av2b3DMTx4xWkV1snPz+edd95h4cKFXHnllTz00EMAFBcXc9999/HGG29gZtx66624O5s3b2bQoEGkpqaycOFCOnfuTEZGBo8++ignnHACd9xxBwAPPfQQLVq04O677+bRRx9l1qxZ7N+/n6FDh/Lwww9HFX92dja33347X3zxBRBKTueeey7Lli1jzJgx7Nu3jyZNmjBlyhROPvlkpk6dymuvvUZBQQF79uxh5MiRzJ07l7179/Lpp58ydOhQfv/73wMcjDs/P58hQ4YwYMAAlixZQseOHXn55Zdp0qQJy5cv57vf/S7NmjVjwIABvP7662RmZlbzbyMkZr8mu3sRcCfwBvARMMvd15jZ7WZ2e7jaL4C2wF/NbKWZZYTPHwcsNrNVwDLgNXf/Z6xijdSsMJc9DfV4SaQ2eumll7j00kvp1q0bbdq04b333gNg0qRJfP7557z//vt88MEHfPvb3+auu+6iQ4cOLFy4kIULFx5ynREjRjBz5syDx7NmzeKaa65h/vz5rFu3jmXLlrFy5UpWrFjBokWLoorthz/8IT/60Y9Yvnw5L7zwArfccgsAp5xyCosWLeL999/nl7/8Jffff//BNkuXLmXatGksWLAAgJUrVzJz5kxWr17NzJkz2bRp02HfZ926dXz/+99nzZo1tGrVihdeeAGAm2++mYkTJ7J06VKSa2ih0Zi+B+Hu84B5Zc5NjPh8C3BLQLvPgJ6xjK08xxTvIDfl1Hh8a5GEUdlv+rEyffp0xowZA4R+yE+fPp0+ffrw5ptvcvvtt9OgQehHWps2bSq8Tu/evdm2bRubN28mOzub1q1b06lTJyZMmMD8+fPp3bs3ELpjWbduHeefHzgEeog333yTDz/83yz+3bt3k5eXx65du7jppptYt24dZkZhYeHBOhdffPEhsV544YW0bNkSgO7du7Nx40aOP/7QCTNdunShV69eAPTt25cNGzawc+dO8vLyOOeccwC4/vrrefXVVyuNuTJ6kzpC6TIb65vqLWqR2iY3N5cFCxaQmZmJmVFcXIyZ8fvf/x53r/I0zmHDhjF79my2bt3KiBEjgNALZWPHjuW2226rcnwlJSUsXbqUJk2aHHL+Bz/4AYMGDWLOnDls2LCBgQMHHixr1qzZIXUbN2588HNycnLg2ETZOvv27cM9cP7PEdNIbISduVtpYCVaZkOkFpo9ezYjR45k48aNbNiwgU2bNtGlSxcWL17M4MGDmThx4sEfqNu3bwegRYsW5OXlBV5vxIgRzJgxg9mzZzNs2DAALrnkEiZPnkx+fj4AX375Jdu2bYsqvsGDB/OXv/zl4PHKlSsB2LVrFx07hmb4T506tcr9jkbr1q1p0aIF7777LkBUA/jRUIKIsCs7C4CGLfWSnEhtM336dIYOHXrIuauvvprnnnuOW265hU6dOtGjRw969uzJc889B8Do0aMZMmQIgwYNOux6p512Gnl5eXTs2JH27UP/zw8ePJjrr7+e/v37c8YZZzBs2LByE0yPHj1IS0sjLS2NH//4x0yYMIGMjAx69OhB9+7dmTgx9DT93nvvZezYsZx77rkUFxfX5H+SQ/z9739n9OjR9O/fH3c/+KjqSFisbk3iIT093TMyMiqvWI7Vi17mjAUj+fCSGXTvP6QGIxNJfB999BGnnqrxudoqPz+f5s2bA/DII4+wZcsWHn/88UPqBP0dmtmK8ta60xhEhP07Q69ptEjtEOdIRESq5rXXXmPcuHEUFRVxwgkn1MjjLCWICKXLbLRsF/cloUREqmT48OEMHz68Rq+pMYhI+dvY7w1pcYz2oxYRUYKIkLwvW8tsiIiE6SdhhJSCHPIa6O5BRASUIA7RrHA7exqlxjsMEZFaQQkiQsuSHRSmaB0mkdoqOTmZXr16cfrpp3PFFVfU2BLYU6dO5c4776yRa9UlShBhxUVFtPLdlGiZDZFaq0mTJqxcuZLMzEzatGnDE088Ee+Q6jRNcw3bkbOZVHOshRKESKVe/ylsXV2z1/zaGTDkkair9+/fnw8++ACgwiW1y1tCe8qUKYwbN4727dvTrVu3g2scbdy4ke985ztkZ2fTrl07pkyZQqdOnRg1ahRNmjRh7dq1bNy4kSlTpjBt2jSWLl1Kv379YraMRjzpDiJsV3boJbmGLbUThEhtV1xczFtvvcWVV14JVLykdtAS2lu2bOHBBx/knXfe4V//+tchq7DeeeedjBw58pBlw0vt2LGDBQsW8Nhjj3HFFVfwox/9iDVr1rB69eqDay/VJbqDCNuT+yUATVprHSaRSlXhN/2atG/fPnr16sWGDRvo27cvF198MUCFS2oHLaGdk5PDwIEDKd2mePjw4XzySWjPsqVLl/Liiy8CcOONN3LvvfcevNYVV1yBmXHGGWdw3HHHccYZZwChdZ02bNhwcBnuukJ3EGH7d24FoEXbjnGORETKUzoGsXHjRg4cOHBwDOKBBx5g0KBBZGZm8sorr1BQUHCwTXlLaEe7PHhkvdJrJSUlHXLdpKSkSrcNTURKEGHFeaFlNlodqwQhUtu1bNmSCRMm8Ic//IHCwsIqL6ndr18/3n77bXJzcyksLOT5558/WHbOOeccXC772WefZcCAATHpQyJQgiiVv4193ohmzY98iVwRib3evXvTs2dPZsyYUeUltdu3b89DDz1E//79ueiii+jTp8/BsgkTJjBlyhR69OjB008/fdiKqPWJlvsOy/jT1XTIW02HBz+p4ahE6gYt9534qrrct+4gwlL255CXrGU2RERKKUGENSvcwd5GeotaRKSUEkRYq5IdHGiidZhEREopQQCFB/bT0vO0zIaISAQlCGBnzhaSzEnSMhsiIgcpQQC7skNvUTdsqbeoRURKKUEAe7aH1mFq2lrrMInUZqXLfffs2ZM+ffqwZMmSal1n/Pjx7N27t4ajq3uUIIhYZiM1Lc6RiEhFSpfaWLVqFePGjWPs2LHVuo4SRHS0WB9QvDuUIFof2yHOkYgkjoEDBx527tprr+WOO+5g7969XHbZZYeVjxo1ilGjRpGTk8OwYcMOKXv77ber9P13795N69b/e3fp0UcfZdasWezfv5+hQ4fy8MMPs2fPHq699lqysrIoLi7mgQce4KuvvmLz5s0MGjSI1NRUFi5cWKXvW58oQQC2J5s9nqJlNkRqudLVXAsKCtiyZQsLFiwAYP78+axbt45ly5bh7lx55ZUsWrSI7OxsOnTowGuvvQaEVn1t2bIlf/rTn1i4cCGpqZraXhElCKDhvmx2JLWiWbwDEUkgFf3G37Rp0wrLU1NTq3zHAP97xAShZblHjhxJZmYm8+fPZ/78+fTu3RuA/Px81q1bx3nnncc999zDfffdx+WXX855551X5e9ZnylBACn7c8lv0CbeYYhIFfTv35+cnByys7Nxd8aOHcttt912WL0VK1Ywb948xo4dy+DBg/nFL34Rh2gTU0wHqc3sUjP72MzWm9lPA8q/bWYfhP8sMbOe0batSc2LtmuZDZEEs3btWoqLi2nbti2XXHIJkydPJj8/H4Avv/ySbdu2sXnzZpo2bcoNN9zAPffcw3vvvQdAixYtyMvLi2f4CSFmdxBmlgw8AVwMZAHLzWyuu38YUe1z4AJ332FmQ4BJQL8o29aYliU72JrSNxaXFpEaVDoGAeDuTJs2jeTkZAYPHsxHH31E//79AWjevDnPPPMM69ev5yc/+QlJSUk0bNiQJ598EoDRo0czZMgQ2rdvr0HqCsRsuW8z6w885O6XhI/HArj7uHLqtwYy3b1jVduWqs5y315Sworx10LXC0m/8ntVaitSn2i578RX1eW+YzkG0RHYFHGcBfSroP53gder2tbMRgOjATp16lTlIC0pifQfz65yOxGRui6WYxBBG74G3q6Y2SBCCeK+qrZ190nunu7u6aUbkIuIyJGL5R1EFnB8xHEasLlsJTPrATwFDHH33Kq0FZGjy90xC/r9TWq76gwnxPIOYjlwkpl1MbNGwAhgbmQFM+sEvAjc6O6fVKWtiBxdKSkp5ObmVusHjcSXu5Obm0tKSkqV2sXsDsLdi8zsTuANIBmY7O5rzOz2cPlE4BdAW+Cv4d9KisKPiwLbxipWEalcWloaWVlZZGdnxzsUqYaUlBTS0qq23lzMZjHFQ3VmMYmI1GcVzWLSaq4iIhJICUJERAIpQYiISKA6NQZhZtnAxmo2TwVyajCc2qAu9gnqZr/Up8RR1/p1grsHvkRWpxLEkTCzjPIGahJVXewT1M1+qU+Jo672K4geMYmISCAlCBERCaQE8T+T4h1ADNTFPkHd7Jf6lDjqar8OozEIEREJpDsIEREJpAQhIiKB6n2COJp7X8eSmU02s21mlhlxro2Z/cvM1oW/to5njFVlZseb2UIz+8jM1pjZD8PnE7ZfZpZiZsvMbFW4Tw+Hzydsn0qZWbKZvW9mr4aP60KfNpjZajNbaWYZ4XMJ369o1esEEbH39RCgO3CdmXWPb1TVNhW4tMy5nwJvuftJwFvh40RSBNzt7qcCZwPfD//9JHK/9gPfcPeeQC/gUjM7m8TuU6kfAh9FHNeFPgEMcvdeEe8+1JV+VapeJwjgLGC9u3/m7geAGcBVcY6pWtx9EbC9zOmrgGnhz9OAbx3NmI6Uu29x9/fCn/MI/fDpSAL3y0Pyw4cNw3+cBO4TgJmlAd8ktPlXqYTuUwXqar8OU98TRNDe1x3jFEssHOfuWyD0wxY4Ns7xVJuZdQZ6A/8lwfsVfhSzEtgG/MvdE75PwHjgXqAk4lyi9wlCyXu+ma0ws9Hhc3WhX1GJ5ZajiSDqva8lfsysOfACMMbddyf6lpfuXgz0MrNWwBwzOz3OIR0RM7sc2ObuK8xsYJzDqWnnuvtmMzsW+JeZrY13QEdTfb+DqOt7X39lZu0Bwl+3xTmeKjOzhoSSw7Pu/mL4dML3C8DddwJvExo7SuQ+nQtcaWYbCD2m/YaZPUNi9wkAd98c/roNmEPosXTC9yta9T1B1PW9r+cCN4U/3wS8HMdYqsxCtwp/Bz5y9z9FFCVsv8ysXfjOATNrAlwErCWB++TuY909zd07E/p/aIG730AC9wnAzJqZWYvSz8BgIJME71dV1Ps3qc3sMkLPT0v3vv5NfCOqHjObDgwktBTxV8CDwEvALKAT8AVwjbuXHciutcxsAPAfYDX/e7Z9P6FxiITsl5n1IDSwmUzoF7RZ7v5LM2tLgvYpUvgR0z3ufnmi98nMTiR01wChx/HPuftvEr1fVVHvE4SIiASr74+YRESkHEoQIiISSAlCREQCKUGIiEggJQgREQmkBCFyBMzsZ+FVWT8Ir/jZz8zGmFnTeMcmcqQ0zVWkmsysP/AnYKC77zezVKARsARId/ecuAYocoR0ByFSfe2BHHffDxBOCMOADsBCM1sIYGaDzWypmb1nZs+H15Yq3Wvgd+H9IZaZWdd4dUQkiBKESPXNB443s0/M7K9mdoG7TyC0ntcgdx8Uvqv4OXCRu/cBMoAfR1xjt7ufBfyF0Bv9IrVGfV/NVaTa3D3fzPoC5wGDgJkBuxKeTWgzqnfCq9A2ApZGlE+P+PpYbCMWqRolCJEjEF66+23gbTNbzf8WcStlhPZ8uK68S5TzWSTu9IhJpJrM7GQzOyniVC9gI5AHtAifexc4t3R8wcyamlm3iDbDI75G3lmIxJ3uIESqrznw5/Dy3UXAemA0cB3wupltCY9DjAKmm1njcLufA5+EPzc2s/8S+mWtvLsMkbjQNFeROAlvsKPpsFJr6RGTiIgE0h2EiIgE0h2EiIgEUoIQEZFAShAiIhJICUJERAIpQYiISKD/D9QOfl+ZMrvTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(np.maximum.accumulate(train_y), label='Active Learning')\n",
    "ax.plot(np.maximum.accumulate(seed_data['bandgap'].tolist() + random_selection['bandgap'].tolist()), label='Random')\n",
    "\n",
    "ax.set_xlim(ax.get_xlim())\n",
    "ax.plot(ax.get_xlim(), [search_data['bandgap'].max()]*2, 'k--', label='Best')\n",
    "\n",
    "ax.set_ylabel('Max Bandgap')\n",
    "ax.set_xlabel('Step')\n",
    "        \n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blessed-primary",
   "metadata": {},
   "source": [
    "We can see the active learning approach tend to do better over time but that this is not guaranteed. Each time you run this notebook, it may do better or worse."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}