Tema 4 - Aprendizaje Profundo por Refuerzo/code/dqn/01. DQN_lunarlander.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Learning para _lunar lander_\n",
    "\n",
    "[**Juan Gómez Romero**](https://decsai.ugr.es/~jgomez)  \n",
    "Departamento de Ciencias de la Computación e Inteligencia Artificial  \n",
    "Universidad de Granada  \n",
    "This work is licensed under the [GNU General Public License v3.0](https://choosealicense.com/licenses/gpl-3.0/).\n",
    "\n",
    "---\n",
    "Ejemplo basado en:\n",
    "> Udacity (2019) Deep Reinforcement Learning Course. Disponible en [GitHub](https://github.com/udacity/deep-reinforcement-learning/tree/master/dqn)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comprobar si está ejecutando en Google Colaboratory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if IN_COLAB:\n",
    "    %cd \"drive/My Drive/ECI/eci2019-DRL/Tema 4 - Aprendizaje Profundo por Refuerzo/code/dqn/\n",
    "    render = False\n",
    "else:\n",
    "    render = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explorar entorno virtual\n",
    "\n",
    "En este ejercicio utilizamos el entorno virtual [LunarLander-v2](https://gym.openai.com/envs/LunarLander-v2/) de [OpenAI](https://openai.com). \n",
    "\n",
    "![](https://github.com/jgromero/eci2019-DRL/blob/master/Tema%204%20-%20Aprendizaje%20Profundo%20por%20Refuerzo/code/dqn/lunarlander.gif?raw=true)\n",
    "\n",
    "En primer lugar, vamos a explorar cómo funciona este entorno."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Crear entorno:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(\"LunarLander-v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cada estado es una tupla de 8 elementos. Los dos primeros valores corresponden a la posición del módulo.\n",
    "\n",
    "El agente puede realizar 4 acciones:\n",
    "\n",
    "```\n",
    "    NADA = 0\n",
    "    ACTIVAR MOTOR DE ORIENTACION IZQUIERDO = 1\n",
    "    ACTIVAR MOTOR PRINCIPAL = 2\n",
    "    ACTIVAR MOTOR DE ORIENTACION DERECHO = 3\n",
    "```\n",
    "\n",
    "El objetivo en [LunarLander-v2](https://gym.openai.com/envs/LunarLander-v2/) es conseguir aterrizar el módulo lunar en el espacio designado ($+200$). La puntuación que se obtiene por moverse desde la parte superior de la pantalla hasta la zona de aterrizaje está en torno a $[100, 140]$. Si el módulo se desvía de la zona de aterrizaje pierde puntuación. El episodio termina si el módulo se estrella ($-100$) o se detiene ($+100$). Cada para que contacte con el suelo aumenta la puntuación ($+10$). Activar el motor principal supone una penalización ($-0.3$ por frame). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Box(8,)\n",
      "Discrete(4)\n",
      "(-inf, inf)\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space)\n",
    "print(env.action_space)\n",
    "print(env.reward_range)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Box(8,)\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space)\n",
    "print(env.action_space.sample())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementación de un agente con comportamiento sin entrenar utilizando la clase [`Agent`](dqn_agent.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "\n",
    "state = env.reset()\n",
    "for j in range(200):\n",
    "    action = agent.act(state)\n",
    "    if render: env.render()\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "        \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algoritmo\n",
    "A continuación se proporciona una implementación genérica del algoritmo Deep Q-Learning (DQN) y su aplicación a [LunarLander-v2](https://gym.openai.com/envs/LunarLander-v2/).\n",
    "\n",
    "Se considera que el entorno [LunarLander-v2](https://gym.openai.com/envs/LunarLander-v2/) está resuelto cuando se obtienen más de $200$ puntos de media durante 100 episodios consecutivos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episodio 100\tPuntuacion media (100 anteriores): -147.02\n",
      "Episodio 200\tPuntuacion media (100 anteriores): -92.17\n",
      "Episodio 300\tPuntuacion media (100 anteriores): -10.85\n",
      "Episodio 400\tPuntuacion media (100 anteriores): 101.88\n",
      "Episodio 500\tPuntuacion media (100 anteriores): 136.18\n",
      "Episodio 588\tPuntuacion media (ultimos 100): 200.04\n",
      "Problema resuelto en 488 episodios!\tPuntuacion media (ultimos 100): 200.04\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydeZwcRd3/P9+emT1y33dCAiSEAAFCSECRWwgCRvECRBFR1Aeex+Pxh6CIeIIPKoogh3IrKIoicgcMR4QA4UggJEASEnKR+9zNXjP1+6O7uqurq7qr59iZ3a3367Wvnenprq6Znf1+63sWMcZgsVgsFospTrUnYLFYLJauhVUcFovFYkmFVRwWi8ViSYVVHBaLxWJJhVUcFovFYklFttoTqCRDhgxh48ePr/Y0LBaLpUvx8ssvb2aMDdW93q0Vx/jx47FgwYJqT8NisVi6FES0Ku5166qyWCwWSyqs4rBYLBZLKqzisFgsFksqqqY4iKiBiF4kooVEtJiIfugdn0BELxDRMiL6CxHVecfrvefLvNfHV2vuFovF0pOppsXRCuB4xtjBAA4BMIuIjgDwcwDXMMb2BbANwPne+ecD2OYdv8Y7z2KxWCydTNUUB3PZ7T3NeT8MwPEA/uYdvwPAx7zHs73n8F4/gYiok6ZrsVgsFo+qxjiIKENErwHYCGAOgOUAtjPGOrxT1gAY7T0eDWA1AHiv7wAwWDHmBUS0gIgWbNq0qdJvwWKxWHocVVUcjLE8Y+wQAGMAzAAwuQxj3swYm84Ymz50qLZ+xWKxWMrCuu178O+lG6o9jU6lJrKqGGPbAcwFcCSAAUTECxPHAFjrPV4LYCwAeK/3B7Clk6dqsVgsIT563Tx88fbKFBr/9z2v4rN/mF+RsUuhmllVQ4logPe4EcCHASyBq0A+6Z12LoB/eo8f8J7De/3fzO5CZbFYqszm3W0VG/tfC9fhP8tqb31czZYjIwHcQUQZuArsXsbYg0T0JoA/E9FPALwK4Bbv/FsA3EVEywBsBXBmNSZtsVgsKvIFhozTM/J1qqY4GGOLAByqOL4CbrxDPt4C4FOdMDWLpcuys6UddRkHDblMtafS42jPF5BxKvO5b29uw4BedRUZuxhqIsZhsVjKw9QrHsdHrn222tPokXQUKuc5P+RHcwC4ymn2dfPwzNvRjNFCgaEjX6jYHESs4rBYuhkrNjVVewo9ks4Q2pt2tWLhmh24+G+LIq+de9uL2O/7j2LTrtaKz8MqDovFUhIPLFyH55ZvrvY0qkbWi2u05yufq5P3rBoeSnl51TZs3NUCAHj2nc3IFxhWbqn8wsEqDovFUhL/c8+rOPv3L1R7GlWDB8Q7CpW3OJrb8gAAx7vnJ254Dqf/dl7onG1Nlcvy4ljFYbF0Mht3tmBVJ6wKLZ0DVxzv72ip+L2a2tymGg4ReDXChp1h19T2Pe0Vn4dVHBZLJzPjZ0/imKufqvY0Enl7wy5051KpTbtaMf6Sh/DSyq0ljcMVx8d/9xxWb20ux9QABG4pDmMMza2exUFAS3vYwuGd+3Y0W8VhsfR49rTlscdzUXQW81dswUnXPIO7X3yvU+/bmXCFccuz75Y0jli78beX15Q0Fuet93dhn+8+HDrW2lFAs2Bx7GkPfydyGVecb99jXVUWS49n/8sfxYFXPNap91y52XWlLVq9o1Pv2xVYvbUZb2/Y5T/PCopjd2uH6pKi7sHpXefWhrS050MxDq5EANcaafeyurZbi8NisQBRt0UxLNu423gcx/N7MHRfVxX3wqXdnOFD/zcXJ13zjP/cEQa4Zd67+I4iVdaERWu2Y/b1/8GetjzahNTefo05AMCabXvw+tod3j2B6+cu98/pKDD//Wzf0462jsoG6q3isFh6ACs27caJv3oav5rzltkFniwsR03b7tYOLN+0O/nEEskXWCqBuWabu6pXKY7nl2/B+h17jMbJSm1G/rJgtdF1HfkCdrUE1sEP//UmFq7ejjfW7UBrR+CG6tvgNvg47bfzcMu8wK12j+BGFN/33KUbMemyRyqaIm0Vh8XSiVQr2Mwzbxas3GZ0vm9xlGG6n/39fJzwy6dLHyiBL9z2IiZd9ojRuS+t3IorH1mqfK09X8BZv58fSjEuFBjmvrUxJOg5TpH9qS68+xUcdMXj/vOM95kXJAXYtyEXuXZXS9glJp7P3Vnz3rGKw2LpFjR1cpCbk9blxBUcYwyL1+3AQ4vWF33vhWs6J07yrEJQbtjZgj88uyKisN9ct1M7DreO3hPiDHfNX4XzbnsJ181dFjlftjhMeWyxu4fHc8vceXPLZ/mmJnznvtf987jFIbJF6sirKvpzKrhBqlUcFksnolqxlkJTa4c2bvHiu1v9zKG0/vwWbwVbYAynXjsPF979Sqf1QUoLY0yZzbRozXbM/NmT+MlDS0JKAAj3lSKEPxSuVEYPaPSPrd3uuq140gC/L2Bucdz49HJ85S53346tQpHe2X94Aa0deV/Q/+jBxaHretdFFUeb9Lf4+O+eAxBWMpXcWNsqDoulE5FdDGl4XVq55wsMB/zgMVz+zzeU53/6pufxqRufDx2ThSTnC7e9iF8/8bb/vNVL9WwWLKTlm5qwrakN/1lWHhfIso27cNbN80tWpvOWbca3/7rQf84V6Xm3veQfe2dDOMZSEBTHQ6+v99t2AMCqLa6SGdQ76EbLhbDYVqTVU66mFsdVjyzFY4s34OVVWzHtx3NCr21vbofjSWO5PkNM973r/Bk4Y9po6BjWt95oLqViFYfF0om0tBfvqjr9unBrCZ5++deE2oGW9jwuuvsVAPpV6FNvbcKvn3gHC1ZuxetrdvjzfPzNYEvUxet24LbnVuKzf3gB/3g1fb2C7C76zn2v4/kVW7CwxJTf3ZIy5vUNJLzZL925IGQtyCv2y/4RKN8tTW48SKyd4Qq3XbjuyoeXADB3CQ32FNH/PRpNUNja1KYdR1RMveoy6N8YjXlwhvdrEOZcOazisFg6kbgspYv/thBPK9pl6/DdTwnn/XvpRmzzcvuTZNwnb3wep183L7LqBVxria/U42IEOsT3/vzyLXh5lRuoFwXhlQ8vwSdveC712CJc4Gck6bZ5d9CaQ7b8CoJS27zLdSPtFCwhx7c4gs/ljudXAQCyGTMRPXKAK9RfeDdaqb41pr+UaHHUZzPYb3hf7bmixVHJNAyrOCyWTqQQk6Z074I1OPfWF1OPlaQM8jH+fB0qy6ijwPwgO1cscVli97+6Ft+/P1jJi+/9rN/PVx6/6ZkVWLBqG/a05TH+kodww1PLkS+wkBA3nXsm5oPZIfVzahTiCFzB7BTOUbmqOKr7FBQrhLgeiFub2tCh6a5blw3EdEPOwZkzxqGXVxR44Oh+oXNFi6OS3Qas4rBYOpFypuPmeXA2QXOIgtk8OB4IHR4kLhSYbzVw4RxXUPiNv7yGu+avCuarOVelTLc2uyvwO59fiZ889CamXvE4WtrzYIxhw874ZoI8LkPSmxWf7pQURy9hx0SuOJra8n5CAP+M2xUJAqrtYvf+7sORvlXiZyqzrbktVAkuMlDY+a8+687ziW8dg2vPOhTHTBoaOnfsoF7+Y7klSTmxisNi6UR0crYYhcJXtUm6oBhdJbqq6r0Vb0eB+ffkWVf5FINf+fASvLwq6qZRfSZ5b/WdcQj3vuQW1LXlC7j9uZWY+bMnQy0/ZLjAdGKk2y6pNUijt4JnjGHz7jbfNbVg1TZ89Lp5aPKaC8oFhozp9xl/YOE67GnL+0q21ftMJ4+Iupq27G5Tpmr/74cn4cLj9vWf1+fcNzVqQCM+evAoX5Fwxg/u7T+2FofF0k1QuTCA4lqKBJv6pLE4zEyOVkFAcldJgTH/nlwYptmC4o7nV+HMm+dHjrd1FHDDU8tD7jG+t0XWIX8uBOC55VsAxO9yyAVm9HMJnuelid/+3Eos27gbG3a2YndrB/Yf6bqArnhgMRat2YEX3nXvK1sc7Xm94li8bgcOuuIxHPXzfwMAWjvyOHvmODz6jaMj57Z05NGs6HP13ydM9JUagIiiEN1YQDgTrLmCiiOaIGyxWCqGTj8U09ojbxgdF1fJppk2ohDnXVfzBebfkwvz1dsCd8zctzbiuP2GxY7bT1EF/ccXVkUKDLmyyzjk11xsb27HHC/LKy5W1NKex8rNTX5arQpVPOGka57GmTPGAQAOHz8Ii9ftjMR65BhHS0ceWY1pk3EcdBRcC2Z7cxta2gtokAS/OJ8Wg3YpDbnwveolxSHWcTR3R1cVEY0lorlE9CYRLSair3vHBxHRHCJ6x/s90DtORHQtES0jokVENK1ac7dYikUn8OIEoQ7TrCrRBaIyOFQZPaLFkcvwHe6CRnpcoIrN/sS6CR1D+kTrDPiOdUvWB5laQY1EIKI+9H9z/ceihSZ/cs1teRz7i6di55EvMAzoFVZiBQbc/YLb/+nQcQMAiKm97jnc4uCfSUt7XmtxiMlWh/xoDna3dkQEP6cjX0C7geKok1LFssLzh/7nKIwd1AvfPmmSO7du6qrqAPC/jLEpAI4AcCERTQFwCYAnGWMTATzpPQeAUwBM9H4uAHBD50/ZYikNnYIoxVWV5H7aIwRdVWfKxWju2IEQyzgEh1w3G79na5Gr2SF96yLHuEXxwMJ1/rEWP06hfm/i59UhfXa6ILN8z4PHDMCD/31U5LWMQxjqpbVyt5dcx/H1Eya682wraBWH6i/KXU3XnX0oDh7T3z/eXmBoN/D7yX9rMaPrgFHueBcdPxHnHrlXKFBebqrmqmKMrQew3nu8i4iWABgNYDaAY73T7gDwFIDveMfvZG4UcT4RDSCikd44FkuXQO+qKiXGEX+eWLNgGuMQBbNDhKznduGuKlWdhwmDe0ctDpUS4v55XVV2SHFIcQfdFq7iW88XGLIO4cDR/aPnAejlpefK75O7/XrXu6+/u6UJLe15NOQcEMKbK6n+pNziOG3qKEwdPQBHXz3Xfw+qVN8kdJ/PD2cfmHqsNNREcJyIxgM4FMALAIYLyuB9AMO9x6MBiP2K13jHLJYug9ZVVYQcDuo4eCfbYGxRsIo1C6YxDnE+GYfgOO79+D3iUkvj6F2fjSQIbN4ddZU1+0V8JoojPJ5JC/eOAtNaMwzw6yTkCnMu3Pt4iuPcW1/Ec8u3YN9hffCD06dExsllCKdOHYmZEwYBABqEtN96wW3V1lEoyurUfT6VpurBcSLqA+A+AN9gjO0UV0SMMUZEqT5NIroArisL48aNK+dULZaS0aXdpklr9a+R0nHFIcTsnx2KQrbEsYXBMo5rceQL0ayqtDDGIu91k1DRzeHBcq3FIYwhu3iWx2Rc+dcXCrE9phpz4SC2HOOQO9YSCGfOGIdtze34+aNL/Xu05xkmDevrZ3iJWVBiYFuuuTh+8rDYnlQc06r1clNVxUFEObhK40+Msb97hzdwFxQRjQSw0Tu+FsBY4fIx3rEQjLGbAdwMANOnT+++25dZuiQ6y6IYV5VscYhj6BSHqc0hrn55jMNVHO6x1oRArk6x5AssYiGoNl/i8Q6dVdBRBotDt1onBK4o3X371IcD63woUTFwN1dDzsFQLylAbOhYF1Ic4c/gzMPH4qQDRvjP/3XRUaFGjMF9q6M4qplVRQBuAbCEMfYr4aUHAJzrPT4XwD+F45/3squOALDDxjcsXQ29q6oYi8P9zWWHOIIoTMMxDrOxxXlmiJDNuBZHgZlZHJ/Q9JvKF5hfo2GCzioQPy+5tkJXvyCOxGMcOnrVSRaH9Hrveimt1vtgBwh9t3hgvSGXwbB+ruLYtCuwrsTUXJ7AwOck1mMAwEFj+uOE/YdDpti9QEqlmhbHBwF8DsDrRPSad+y7AK4CcC8RnQ9gFYBPe689DOAjAJYBaAZwXudO12IpHZ1+KMZV5Vsc3nOdq0oU8qZiJhQcdwgOufUUBSE4HlftvljTBDEvFBGaYBTjKELpduQZMjGl5fVZB0T6qnvZVcWnKfaK4nUUjbkMPnrwKPxn2WZ88agJwTUOYeVVp+L0387zld33Tt0fYwf2wvTxg4zeR4+LcTDG5kH/PT5BcT4DcGFFJ2WxVBhtjKOkdFz3uWgliEFdMZDNz80XGJrbOvwgb9x8MkTIOOF0XEC/m6Gqn5M4bhpBbxYcN7NgxPhpgektDiL33F65TPAehWuJoq4s/urwfkHWGK+jqM85GNi7DnedP1N5v4xDaNrjWhy967I4cUrUstBRrRhHTWRVWSw9BZ1lUUw/Ka4oVH5u0VW1p02sHHfPvfqxt3DQFY9r3ToFRXBctDgA4MAfPKa8du22Pdo5q2IccRgFxxXjibv3qegoMGQShG5jnbrKO+c4kT0x+N9gmNidtj1wVcWRywRpvLlsOkXQ42IcFktPROuqKsXi8MdW+/1VdRLPvuPu+/Ha6u3KsTtkV5WXjlsoAJOG90FfjaUCxAfO08Y4kiyO5rYO/ObJdyKvqywpOV1Zp5T4aeK9xTMzDmmzrvoJLiyelCCfK5N1nEBxyJuIJKBrd1JprOKwWDqRUtNxxeujdRzBeaKrSkz15Nfwlhp8T3IZMfjsEIJ0XMbgEMXus83vIdc18NfKGeP4+yuRxEoA0eZ/7r2Dxx15fcU3R7sjX4YULdvJ//21Y/cBECiOJIsjmyHf8kurOKoV47CKw2LpRHRZVaZt1UXhJ2dViWOL7iDReuDnDGh0s3a2KIrvgLAiI8BPxy14aaw6gbVqS5N/DzkzCQCeWLIR18x5O3I8Lfw9qVJ5AbXi+MQNz+FCbwvdOIuDf546xZG0b/x3Zk32u+sC0Z0IZXIZx38fuZQxCxvjsFh6ACovzY497dosJJm8Qgmo0nHvfH4VVHCBy69t1VSAi/NkCFscvK5DxTFXP+VbPjo3yv2vrVMeV6HbsZBbRDq3l9w1lvPQovWY8+YGtBeCrCrV/hhAOHU5bShBPP2g0QNizxUVWFqLw8Y4LJYegMriOOvm+fj6n19TnB10jlVdH2zk5LmqBBl63ytrlOPlfcXhPtfFI2R3kuPwdFzXHRPX84rPsZKrYW4R7W5VKz6VxcH58p0L0NYRVI7//b8+gPM+OD5ynmhVvfqeOhbkI/1Z+cdz6tSRsXMBwp9T+hhHD0vHtVh6IiqP1Jvr9dbGoVLnWlGgB1vHemMr+7GG4UKdu8Z0rh7RVcWY6255Yom7F8a0cQNi9/Pmc0wrBDkNOSexiSK/x9amaLsSQG9xiHDF0Ksu63fDFUmzmpc/e36pyQiiZWZjHBaLJUIxrUVE8lJmECC2HAnOmzyiL8Yp2mrz2EfgqlILaLmSXSyWi3NVifModjWcM8gU4u9dF6Op02yYJCIKXVFJcAsuSW+IVor8Z+VjmCifsMVhYxwWi0WimApxkUJcjEPKuFKlgfJrAleVpqcUY6HYiSifklxVfB7FWhyiMEzav0RXhJjG4gDCrem59RBnVQHAD04/AN/9yGTla77FYSDXc6VYHDbGYbF0f4rZIlYk5KriWVWKsQtMvRrNS8FxrauqwEK7zYUC3QyIMwp8i6PI1bAYE9DtUcHfhy4bLSmuAIQtItEy4HuGmFgL/Bx5FiT9jp1HCTEO66qyWHoAoqDTrfbjyDOVxcGFVzhwrnIVBQLXfa5THAVJcYiKosBY7EpX3C/clH2H9QHgFtD99GMH+cd1WVN5FlaAQNjKkLdYVaFzVd37lSMBmFkLqr1QxItNNs4SlYXJvEWqVQBog+MWSyciupr2u+xRrLzq1JTXu78Xrt6OO59fCUBY1Yo1HoyF9qP+4L6DUZ/N+K25k2IcHQWGPvUZwIs9iwKKFwHq56hvhaJj6pj+uP/CD6JPfRZvb9gVzENncfBYTQGYMX4Q7v2qK+zHX/IQADNXVVbhqhrVvwHjBvcynj83FkqxOEQFlrblSFLblEphFYfF0omU7KryBP7s6//jHwsKAMX7sMiKOuNQJDi+9P1ASIfnyXx3D2Ph3fKS3gN/PY3icIj8NiHiqlvegY/DW6oUhFiMiFGMI2RRRQfRWUx/+lLQrFBXQU8pNIfoqkprQVQrxmEVh8XSiZhmVc369TM4Yu/B0esVUttRuqrCGToOETJEkeC4jnyBhVwo4sI2qco9cFXF30NElL+iINV1vm3nLjeohWfaGIfKpaTztE0fPzByXTSriv82cFUJysJk3iI9rq26xdITMU2qWvr+LqU1ENfnSbY4xNWrQ66QSQoqi2NxAc4QTsctMBa7kpZjLyaI1smQPvUY0CuH7c3t2hbsXKEwxuAo3DX1qdNxo6/r5p+TPldA4aryYxyJ0wgpSlWblthrbXDcYun+lLOOg+MoArRyLybeX8rPqjJoUCuu5EXrQTYCGnJhMcKKcFWJQrohl8Frl5+E0QMa9cF7wXJS3aeUrKrgmPo60T3lf0bS3yVNjEO07FJnVdk6Doul+yMLftPmhhyVq0olu+R0XPJiHKpsJB3iily0Xhhj0SW2QN4Pjifewkd1ruPod/fjykv3Pkqp41C9rkOnHJMaJYpwl2Ix4Qpbx2Gx9ABkOZd221OVxaFWHGFXVcZTHAVf4Cbfyxd6DFJwXF5dh4VX3AZTifeSjul2E2RlsTiCc1RuKRNXm6rBJGBefQ4EuwkWY4zaOg6LpQcgWww6V4wOVYyDC6mC7KoSg+OO6wbh+4+bWDrhbJ/gcb4QjnEM7BXeDa+4rKroMYI+Hdd/r4wprzVRHLo6jrg5Rc/RnOQdNvkIdNv3mmBjHJZuxcLV2/HCii3VnkbNIcv9tIpDFZtQBWijwXHC6AEN2NLUhua2DoN2iOFsLVFAyjrn/83aLySouVJKk1mqzmrSWxxcn+gsDpPguKqOQ8TIVeW9R11WlUmUoxTFYS0OS7di9vX/wWdunl/tadQcsptHV6egQ+WqWru9BXc8tzLScl0O/o4f0hsAsGpLs1GMIxxcD47L1/auy+KCD+0tvB7c0xTluaR35TEhVqO6tDwWh3mMQ9sdt8IWR5rMtXJiFYfF0onILqJyuKo2727FDx5YjPXbW/xjcnA84xDGD+aKo8koxuGn4zI5HTd8ntwtN4hxJN+Do3NVbW3S7FAo7CuiEp5pg+NKvZVGcUQsDi/GkThCEOPoSlRVcRDRrUS0kYjeEI4NIqI5RPSO93ugd5yI6FoiWkZEi4hoWvVmbrEUhyx0dS0/9NfrJT5vJwJ4LUckwTi8X4N3XmvqrKpwOm74WscJd8stqo5DoTnilJu4r0ixMY5i03HD58RnVZl8BH0brOJIy+0AZknHLgHwJGNsIoAnvecAcAqAid7PBQBu6KQ5WixlI+KqUigOVcotJ64AcP2OQHEwqVdVhshv5dGeZ0bBcVEoRtJxBXjGlj9/Xjmeqo7D+FT3HgXgo9fNw9L3d2liHMW5qsR3ZjJ/P76k7nFoVDluLY6UMMaeAbBVOjwbwB3e4zsAfEw4fidzmQ9gABGN7JyZWizxLFy93c9YikOOUahiHHEpunFKZe32PcF9FDEO7nrKFwpmBYCaFbnKVSWKRz52qTGOOKuowBgWrdkBQK10TArpQllnRbqqSKFwgHTpuKXEOKpFtS0OFcMZY+u9x+8DGO49Hg1gtXDeGu9YCCK6gIgWENGCTZs2VXamFguAddv3YPb1/8H3738j8VxZFqosjh172rXX5xnTZqutExSHHONwnMAqaM+zVMFxJo0lKz+HKJLRBaSzIlSCO26K4hzS9JkSEeM2KleZaoyzZ46TxuCfkTo4bqI8TayjWqOmZ8zcv0aqshjG2M2MsemMselDhw6t0MwsPZWtTW34yl0LsKM5EO47W9zHfAUch0kdx+E/fUJ7fb7AtNlqzdJueHKvKr4KzxeYUXC8X4NbnzGoT52UjisrjrCQ91uOpIiOp7c44q812rK1iKyqn338IOmcxNskQkQ4cHQ/XHH6lKKu/+zMcfjNmYeUPpEU1KKNtIGIRjLG1nuuqI3e8bUAxgrnjfGOWWqIPW15NKZs1NaVuGXeCjy2eAMOHLUS/33CxNBrckqmikgdRz7dZk6bNXtsA9FOsnKvKv60I18winEcs99QHDZ+ID5+6GhcP3eZf1yOs8i1BNzVlkaopk0rFRWwsl2JyV4aFW05Yu6qAoAH//tDZicq+KmkzDqDWrQ4HgBwrvf4XAD/FI5/3suuOgLADsGlZakBHn59Pfa//FEsXpe88u6q9Kpz11q72zqKut4kOB7H9uYYxaHMdvIek5v5lMsQ2gtmrqqMQ/jszL3Qqy4bm47rOBRSmrxoL01wPK2rSpy/KLwf+fqH8MtPHWwk9JMsDrMdANVzJf93deosKk1VLQ4iugfAsQCGENEaAD8AcBWAe4nofACrAHzaO/1hAB8BsAxAM4DzOn3ClljmLnWNw8Vrd1Z5JpWDBzKbWwNLIU2PIXmlnzYdNy4AL7fncMgVWwyBYOQdco3qOFRdYBF1t8lClyuOYtuqc+KsItHqEa/cf2Q/7D+yH9Zsa068Z1IdRxqrpZQCwK5IVRUHY+wszUsnKM5lAC6s7IwspVBEj7aqsGLTbvzXn17BPV8+AgN716W6lu+X0NQaWBziPt6MsViBWWodhxzHEJH35844rvArCLUOOcdBe75gZHGE0nGF4LjqWvFQnKtqVP8GrBPShoN7Rc+Nm6F4P9XnbWZxhFuyFDOG1lUl/e5u1KKrytJF8f+Za/y/5XdPLcfS93fhiSUbUl/LLY4mwVXFV9jvbNyNCZc+HHu9LHTTKo49cRaHwhIQLQ3AVQD5AjOykkzTcRkLr7d5rEUWqkTAlFH9lfdSCX9zV1X0dRM3WSajfn/BnBKH0LuqUsY4uhpWcVjKBhcftf6/Ukzbbw6vSG4SXFXtmg6u6nuHn7fEWBAq9sRZHCpXle8y4QrEMU7HFbOiQt1xVdcKx/jnoV7Fa+6lUhwxNkdeE+PgmLjJkpocpulVFbl/inl0RazisJQP73+51v9ZgnRR82s+cOWTuPTvi/xrRYtDzmaK3d5Vei3OglAR66rKy64q0eJwj+Uy5BYACoJ39IBG5Xjiql1UIp+VahnknHnuMiPp8yXoBW3a4HjYVRV93cTNFI5xFFcLoqo4F+dU4/8KRWMVh6VsmHqqFihVvPEAACAASURBVK3Zrs0O+v79b+B/7nm1rPOSCXaoM/+vXrejBfe8uNp/j2KMo11SBrpW4EDUVdXSnk8lXOJjHNH6Ci78RJdVRz4Ijk8e0RcDpP00OKEmid7D/Uf2w/dPDdcbRDanyqtbjpDgOpNR13EoTwUgBceLLQAUFWPRysf9HQ3ke66qmre/i8MqDkvZEf+PVS0yPnrdf/CZm9RFbHfNX4UHFq6r1NTcOWma8DHGIqt2flx+LLqq5GtiW4ZILzW35cEYcODofkZz39OuTwNWp+O675EL5lzGQUfB7VX1gX0G49FvHK3dDKghF9TjZDwJWZchRWFfOGYS56rSKUn18fiWIxz1trMpYxyK80tqOWItDovFDFX6pE6IvrVhV6Wno8V3VUn/1Nc+uQwTL3sE63fsCR3fLVgX/N1s3NXiV4/LFkZ7TMBb/oyaPZfXSVNGYOoYdeA4fL65q8oJrai94LhD6CgUtBsgiYitMDKSgBSvdLcgD96X76qShifoV/HqdFz93MIFgCqLoxwxjsQhbFaVxVIqvlAR/ltkf39ck77OQhccf37FZjAG/Gn+e6HjYu8oLsza8wwH/+hx/7FIu5QWe/3cZTjnDy9gV0t7xFXFlVLGIaNiwNjguFzRTeQrKu5SyTjkB8eTZKtocYh9q4DwCnuUFCPRWRxEKWMc3m+VKy3cciR6rVFWVUKMg48xsn+Ddgx/iEhWVfh3d8MqjhqlUGDY1aJvdldN5r2zGd/7x+uR41yoiH5dubYgzo3TWXDhLc9lcJ96AMB/lm8OHd/eLP4dVFaVZHFIiuTqx97CvGWb8cjr7+P+18JuuN2eyyvjkFFqbmw6rnRfUWiR4KriBYBJrhhRcXBXjpzpdP3Z0zBqQKPkquLpuOHxso4T46pSWRzuoN/68KTIa0lNDk0EdlIdR5CJph9MGxy3MQ5LNbj23+/goCse1+6AVk3OueUF/OmF9yLHVRaH3L47LuOos+AeHdm1w1f8yzbuDh3f6VkcdVlH6T5p75Asjo4Cfv3E23hOUkBPvxPt1rzN+/s2ZB0ji2PVFn1FtGzpZJygay0XfhmHsHl3Kxau3p7oimkQXFWyxcEZ6FkD4p+Vf66iwD3/qAm472sf0FoCqrnwMVWCm4UUR/Ras+K96GMWsmQC955+DP652MpxSw3w4CK3DdeW3a0YlLK6ubOQq6TVMQ7Z4khX8FYJ+Dzl2ARf8cvutO2e4uhbn1WGa2WB3VEo4NdPvAMAWHnVqf5x1aqW79rXrzGXev9xGfnjVzXxy2UIL63crp2PSL3K4vCtSu+3YgyeZSa+9u2T9kNjXSaxKaAI/zvlFHnT+TLEOMR7xu0AGGtxaJbevuJInEXXxMjiIKIzvK1cdxDRTiLaRUTdtyFRDWDSvbTa6ArfxH9I2cKoBYuj4CuO8FzaOlw3kFzgtsWzCvo15pQWh+wiautQv0eV/Nmws9UduyGHX3+mvK2xiUhIBIi6XdJYHHJwnP/2V+qCSs3no9u5cgGrE7TqAkBE5sxJinGkbXce13KkFFdVdzU5TC2O/wNwOmNsSSUnYwlQuX1qDVkJqMSlHEcopuV2ueFyXnZV+RaH9EbWbnOzrAb2ykV8/J+56Xm88G54E0tdHUeczuzbkMXMvQdj76G9sWJTU9JbMCKjWFGb7IzHCW09q9mwSCVUOwqFiCDmc0kTHOcftVhPwikk1HGkLUJVKTQ5lVl5ncLFpXq9u2H6LdpglUYn438Ra/ebJ7to/Mpx4ZDO4jDxQVcKpgmOtwmuqnZhzwreaZUhKiBkpQGEA9h3C7Gg5lZ9DUa/RnURXik4FFgCfq8qsXVICutP9/fyBbRUxxFRHE6S4tBbHFmFVBfdenE64mCDFGfd/flbViku+Tpde5SeHhxfQER/IaKzPLfVGUR0RkVn1sPpChaH7KLxe1VJ6bh/eHYFPn/ri+41Cv93ZyOm1Ipwi6OjwDDxe4/gqkeXAgj28i4UTLZqCmozAOC7QvbZrha94ujbUP5wo6NwS8XtqxGHHBznIztRvYH2fCGygg9W7+rxVV8HrrhVgrsjIcYBAHO+eTT++KWZ6htKxPWqysT0pkmKpdTy/28pmH5b+8HdA+Mk4RgD8Peyz8gCIPinqeXvnezqUafjMvzkocBYzfs5/pWfnw6+0tZlVXHueeE9XHrK/ljjuap4xXUSuiK9XZ1scWTEGIfDXVXBB2/S6JAjp+MGMY6oC6tDYXFwdAuGeIsj+lo4OK6e88ThfdUvGM6Lv+cYg0N4/5pxjWfQtTBSHIyx8yo9ka7Mmm3NGD2gsSKr6FpuGNheYLjp6eU4bK+BmD5+UKA4JItDhGdVpdkdLi0vr9qKtzfsxlkzxilf53Nas20PNuxswdNvbcKWpja0doQFft+GHFra89i0qzV0XRLiJk8iurocIqBPXSUsDuGxIjiexlWlS8fl4506dRR+/+y7AFwXpk5xaCvHFYt6rtiS3JrFdDk2GUPu86Ui6dY1/O9bEqZZVWOI6B9EtNH7uY+IxlR6cl2Bl1dtxVE/n4u/LlhT1nGrn3uUTEe+gCsfWYpP3vh86Li4kpXTb4tpMKjjhRVblNvUfuKG53Hp36MFipxWz9L4y4LVmPmzJ3HxfYvw80eXRiyOPvVZ300FcIsjeV7Nmm1ld7d2oC7r4H+lgrbedVmj3kr9U1olDgl1HIrgeCqLw7tevob/GQ8ZOwB3nT8DgGtx6P68urcZ13IkKaBfDtlcbFt1jtbi6KaawzTGcRvcPb9HeT//8o71eN7e4BaLvfLetrKOm+afGgCeW77ZLybrLOQYAXdjiLEPOX4exDhKv/9nbp6PU6+dl7rCXldoJ1du967P+G6qEf0aUDDcq7tJ56pq6cCBo/ph0oiwCyUu+Cq2PD/z8LGJ9xYRBR9/KLp90pTUBFukeuMp7sGVU3teb3Gkq+OIztn02rQUuwNgN9ULiZgqjqGMsdsYYx3ez+0AhlZwXl0GXafVUuH/NCaCKl9gOPv3L+CcW14o6xySkK0JPlUxcKmzOIrNqip4cQaxLfvKzepqal08YnerWtG0dRRCMYA+DTms8yyOcYN6lWxx5AsMuYwTWSGLQuuTh4UN+bGDGvGdWZOTb6og46ZVBY8RVlLKDZk8Rkn9mTJSFFyOcQBBTKA9X/DPl4P+/JzGXCbUgyru6xCnWOU5FItqiDTD6r5r3VWxmCqOLUR0DhFlvJ9zAGyp5MS6ClxGVirYaxKM5cplyfrOrcmMZlW55AVlEY1xuM+3Nbdjzpv6rVt3aqyIvb/7ML75l9ewZH3QXfe9rWrFoavE3takHrujwEL9mRpzjm/NDOpd5265qp0x8P9O3g9AfAdbcXMljvjd+dox+2DJj2bhpCnDAUjuqTjhqmwLLt6Dp+MG//Ly36bOcwn96tMH45mLj4vMW4V4OOsrjqAAcM43j8Ffv3pk5PzD9hqI1y4/STiuCo7zRUa8mCrH/15c9Xnc3zxpwdjT03G/CODTAN4HsB7AJwHYgDkCwV6OVU94XPe3SfySK47OjovIhW6qNFe5VkJUKl++c0HoNbGoa+oVj2vve/9r60Ktzy+8+xXMfWtj5LyWtqjiaO3Ih9qky/SqCxRHR55hjzdG7/qst1d3+P0cPHYA3r3yI1h51am+tRDXwZZI0W485FIiNNZlsO+wPgDCjQZFbvvC4ZFxZQqFQPjylf5AYZUvv5eLZ+2H/3fyfph9yOhQ8R8gBMf5/RT35XNtbsv772lE/wYcPn6Qfw7/P5FjOnG9qpJdVbEvG6EaIyju0/9n8cvkM/zPvXvqDTPFwRhbxRj7KGNsKGNsGGPsY4yxaJe7ToCIZhHRW0S0jIguqcYcRAqFyn5BTFxVXBZ3dpeSaMsR97m4ko1YHDH7c6fpnMsznTiLVkeD5KpOsjprg9MoCOrWjgL2tOdRl3FQn/M2QJLOzwqbJfEgbpzFIe4DHhyLnsdX+LrAsHxc2TNKUOz8HkfsPdg/JruqetVlceFx+yqtC11wXITv37GnrSMxCC57n2J7VSUFxyvkDzLrd+X+1gfHyzihGiL2L0JEF3u/f0tE18o/nTPF0HwyAK4HcAqAKQDOIqIp8VdVFi7ryh/j8KyIFBZHZ6MLMosCK02vqqQGiOLK78pHloZea+mICuvTfvss5r0T7lB7/dxlsfdoFNJiWzvyaGnPoyHnIEPuXt2y5hCFLPfF62IcHPm7onJn8HPqsup/Udnvr/r2teUL/veHB65n7j0YYwa6AXf/z+S9FvcV5vdT7cfB8S2O9nxicDzqrlO5qry5V2nZniqrylaOh+CVWwsAvKz46WxmAFjGGFvBGGsD8GcAs6swDx/dpkClwr+GRhZHlRRHi7SiVwXHdTEOFfJr9y5YHXoep3Ra2wOl05Bzv9abd7fhm/e+FrrurvmrtGMAblzDH7OjgJb2PBrrMsg45MU4wnMQ61F4jKBJU8cBeBZH5JjqPPd3TiM0ZfeN6usnWnfi3hI//fhBABStwLWzTm6tAQD13mfHWHJPKhOryzSrqlKoquJlGrKuspykKTbskRYHY+xf3u87VD+dM8UQowGI0mSNd8yHiC4gogVEtGDTpuj+B+VGtw1pucY1szjKe29T5PTVIDgeF+MIP39/RwsuuW8RWjvyflU55+K/LcL4Sx7C315eoxxLRLQ4xLjApl2t2Oe7D0fuW69ZyfdpCGIAre2uq6ohl0GWKw5pCuLKP3BV6S0ON8YhC31FKmhCuws5RqBa2bbnC8oiOq6M5M8kTsjJq37VqeGNn9Tj6OpV4hZenWlxiIpRbiWvYmDvOvzpSzNx/WenVXpqNYVpAeAcIhogPB9IRI9VblrFwxi7mTE2nTE2fejQymcM+xZHmb/c/AtsYk2U2oL9qbc24v0dLamvkyut/Z31BAWQT9gB8GcPL8GfX1qNxxdviDZN9Pj9MysA6LvOAq6Q/+P8VbhmzttKpfDzR5fixXe3oi7r4AP7DMbeQ/sox+lbH7iq2vIF7GnLozHnWhyqGIco1DIOoS7jxPakUlociv9C/n3SpaLGZWZxhvSp9xcVveuFbWA9BSfHOOLcKn5b9ZivGl99q+YXPZ5sMfn3TvjfKub7f+sXpofvr3jvpl6ED+47BP0a1MWZPb0AcChjbDt/whjbBmBYZaYUy1oAYhXUGO9Y1fBjHGUetxIWx7rtezD+kofw5JINYIzhqkeWYtnGXfjCbS9h9vXzUs9Rt9VpRygdN/yarEh4MHpXS0esK6pQYLEuoJaOPC67/w385sl3UJ+NZiLd/MwKfPqm59HWUcDh4wehTiOQxbqD1va8b3FkNBaH3DqlIeeE9iiXIajcNPGrbZV7SJalsoCaMWEQjpo4xH8uCjY5XhGanG7eUhBY9ZfKZYLAvy4JQu+q6jyLY0ifehw/eXjoGL99yGoqR31IySPUJqaKI09EfuMfItoL1emK8RKAiUQ0gYjqAJwJt6K9alSsAFAaPw7TnkMLV7u6/68L1mDT7lbc+PRynPMHt2st31AoDboYRzgdN97i4IL6u/94PVYx/PBfi3HElU9qX29tF11V8V/rxrqMUrkAbpsRf0we4+CuKqaIcUhCrbEuo61BAcKbK3FUAor/TUX/vrgqlq+RR/jAPoNDz0OKQ+eq0s7azFcvvjexVYtI2opy03vH8aWjJoSeb94d/a4P61uP//3wJNxx3gxhTu7vUiz6bmpwGCuO7wGYR0R3EdEfATwD4NLKTUsNY6wDwEUAHoMbuL+XMba4s+chzQkAkGJ/nFSY6ATTL3bed6sFx0rZylUf4zDPquojrPAfeX299l53vxif/S3ORVf7wGnMZfxArow4H56O6wbH3f3G5W1lI4ojl9HujAi4gsRkoc8796r2ouDjxA0iC2LRkuJjRmMcccI7nI470qssr8vEf9Yy3AVnkiDASWqImfT1v+y0IPGyLuNEqvMB9/399wkTMX5Ib/9Yb28RoWshYzKncifN1Aqm3XEfJaJpAI7wDn2DMbY57ppKwRh7GMDD1bi3ikKFviCBq8okq8psTFXqcJraCRkxkwlQb5AkPr5+7rKQZQC4HWg5jXV6IZRxKFYgi9ZPg8aa8O+Ty+iD44LFsbu1A81+jMM9Jn9esuJIUlpKAanKiOIWh2GMI0kQiwqRt1WJNCxU3in8Gr/kurOmYe5bGzFucK+Yq6LoXFVxSisxxpHi/ot/dLJxZ+YhfeoBAFtL6AHXTfWG8X4cAJAHsBFAA4AprlnKnqnMtLoOca4qxhjeXL8TYwf10gbP9PDguPkcEkcUUoe520POZEqDHBznhIPjweOrH3srcm5jgqAF3H8+d5Wst462CP/cOmuC0yC4qn5z5iEY2qceZ//B7fMl/51WbGrCoWMH+tlNclGhnCqauLEPyGgxIPb0Uv19I4oj4XlOsQ1smqwqP8bhfS8H9q7DGdP0DbLF5owifKHA3ZK96zJoasvHWhzlTDxJs3XukD51AEpUHEVfWduYZlV9Ca576jEAP/R+X1G5aXUd4npV3f/aWpx67bzEojMV6SwOQ1cVF0bCXOMa3akQM5siriq/jkPvqopDV3HNWPKqc/32ICtMZ01wRIujraMQUjR9FDvxNdY5voL4v0fDyk8WanGZX+75ZivkDkWMIzROQh1EnBLgwjNN/c+QPvWYNLwPrjzjIKPz533nOOXx0w8eiXOP3AtfPGo8APitVeK+h5XcuyWOwZ7FURLd1OQwVb9fB3A4gFWMseMAHApge/wlPYO4XlU84Cy3x+Cs274HyzbuUr4WBMdN5pB8jjiWI2TqxLUAkWlq7cDE7z3iP09rcajnFLzeFFP/kEvokBq2BOLPdWMcrsXR2lEI+em5Qsk45Kfm1mcz2lWvLNh5Y8XPH7mX8nzTSuKkLsI6C+OIvd2+UHGWT9Z3Vclj6ueTyzh4/JvHRLKRdOhcT/XZDH44+0B/HF44tzEmOSPJiqtU/euAMuzK2D3VhrniaGGMtQAAEdUzxpYC2K9y0+o6BJXj+td0fOCqf+PEX6m9fUHLkfJlVQV9tUhpHSQhZwv9cX44YO0ro5gCQBnx/el2zntrwy5s3h11F+iUiU6hcRrrHF9B8JYiHFFIcQGbzZB2+9Bvnxz+N+BtWI7ad4jqdFc4G/y52v3geDgLa9YBI9C3Phu1MPzf5L0P9/nkEX0jG0BpXVVVEHOXnToF5xwxDicfMEJ7juMAv/7MIdrXzXaCT085XGTd1OAwjnGs8QoA7wcwh4i2AVhVuWl1HeJ6VaWpxYhcK40fPwdDxSEoOf44TWxc15uKE6Tjiq6q+GvE+8c1B5Q5fPxAfGfW5Mjug4C+voRTl8ngnCP2wj9fW4tTDhqJ4X0DlwQXFg4FVds5x1EG5r96zD4Y1je8bwX/jHRuDiJ1XYZMYHGE13Y3fu4wAMDKzU3SuOHfXAE++o2jI4uPnDemnCFWDiH33Y9Mjq1jkenfK4effCze/ZUhwvH7V6NsDLjxnMMwekAjTr8ufZ0T0H17VZlmVX3ce3gFEc0F0B/AoxWbVRcirlcV84Vz8qpbVjxpNnJSCf8de9rxx/mr8LVj9vGFoZgBVkwylanieHBRkFab4PIPvb+k5oAifeqzkdbfHDlzS6ajUMC+w/rjVWE/iK8cvTc27Gzx/ekE8jOpshky3mWQf0Zi+3IRh6ILCWU6bmKMQ52PGygQof5DOpdbUmnjWyZccPQ+ZR9T3sPkrBnjcE9Cena5mHWg3hIyobtaHKbB8XH8B8C7AF4DUNon2k1gvsWhfy1JSKvyxH1XldEcomdd/s83cPVjb+HZZUHWtNgeRV5tAsD8FVuw//cf1WaR8JX8Td6qNzIPxWwrZXHUZzPatZzO4jj94FEAgNEDoxk/l35kf/z6zEODGhcKVou5jIOdMW1ERHiMQ7c/uKkcSY5xhJ9//YR9AQAj+7vvLbYuwl9ImNdxVBOS2rQM7Ru25qrU49OI2vxES8c0xvEQgAe9308CWAHgkdgreghcAKu+vIUYq0HcH3zTrtaI8A9cVcVZHNub3RXyCyu2COcFrirVsH+cvwp72vOYuzS6KRIQCMU6zUpfNWaqGEcKi6M+52iDpjrFce6Re2H5zz4ScS+JBBZHQNah2P5TIjO9/S76aRSHGF+Kw49x6Oo4JM3wycPGYuVVp/obUcUJLF4AyDOaOLUs5MQ/9SDPmjtodP8qzSYZWwAIgDEWckJ6xYD/VZEZdTHilIO/M5/itUN/PMd/fNwvnsKPP3YgPndENBOn2OA4D3r/7qnlmLn3YBwzaaiv5DKkrg1wM1zW4+2Nu/D+jhZkHAqt7to9gazbI0Jk7yG9sWJzU7qsqpiWIzL1WSey6q7LOGjLFyKtUIJrMolpvarXcxnHz+lP4obPTsPqbc36egFS7RYXJWg5oh5HTlHlp/kCK+Z91mUd/PH8mZgyql94ajUs40Sb43NHjsfkkf0wf8UWvL52R1V6HxlTw59pKRTVKIMx9gqAmWWeS5ckTjnwIyZZTw+8JvVq5ArJIOlJpQTElNi129y+QfwQaRTHwN6ucHxnw24cceWTOPynT4Re9y0OheLYuLMltCr/3TnTkMuQgeIIHqexOFQV2rythk5xmCg8vyUGBa63XIbwzQ9PwqyYzB9O7/osJo9wBfJFx+0bHd9QOvfyNpTqVZfB8H6uhcTbfLjjhM/nCsa0d9pRE4dgUO+wMqy1QO6j3/gQfjT7AABhpZZxKLSTYS1TW59o+TCyOIjoW8JTB8A0AOsqMqMuRtxOffw1E8UhX59mH/EkF5GfQSWk48pTct1X7sH3tjYr78MDvypX1YyfhRsQZh3yO8rGIbrP0sU4oq6qPg1ZbGlq07qqkgoDAYSC4/5GQhkHDbkMzpg2Go8uft94jt8+eT+8vnYHnn472BeGYLaB0uWnT8HE4X1wzKShIHKV4of3D2oouGLoXZfBnG8dI8Qt3Ne7w17Xk0f085VwrB6s4SBHrcaNSsXU4ugr/NTDjXVUdee9WiFwVUVf499nLjzX79iDL97+Ena3RlfWuq++STaP2uIIhKec3SUqCQ5RIOQTFYeJAHYcZB3HIMbh/m7IZdIHx6X/SW5x6BSHkcVB4ToIIMhsihbdJc9T1T7cRMz1b8zhq15GHBHh5ANGhNxPfNz6XAajhPYecQWp2jlq5lpLKLfX7QLr+dqfYXGY1nG8yRj7q3iAiD4F4K+a83sMeUkoi/gbG3nC85o5b+PfSzfioUVRY02+nj/71r0LcePTy/HARUdhV0tHJKNEvvbdzU248/mVaFNUb3MZ7vY/Co9Bwuu6tNs4V5VM1iE4ZFA57r3ekMtgW7N5TyCVxdHbc+/o7mnmqnJ/i0qCxyvKsYp3nPIskP1sPul4vhAsDtJSy0IuTqmZfJyXnzYFE4TOt2n5+ScOQp/69JXktayMS8FUcVyKqJJQHesRtLTncd8ra3DW4eOErCqV4nB/F+OqEp+/vWE3Jn/fLZu5+0sz8QGpKlkc/oI7F+CdjbtDQV6uQ0LpuBGLI1oQJtMa46qSyTiEbMZJrEznt2zIOqkEan0uGhyP664LBIVvcaiC4zyzSVZUukZ+8ZRHkvD4i/wZBBuLpb9PLQs5lQUlby4VxxelPTnS8pnDxyWfpKCWP9NSiFUcRHQKgI8AGE1E1wov9QNgHsnsZvxqztu4+ZkVGNy7zrcmlK4qqGMccfEQ+VqZN9btUCiOaDxDvCcfm6d4ipXjHF3AXIRbIiaxAh7jWLlZ7faS5ysHu5PiIw25aB2HqNAOHz8QKzY1YUtTG279wnS0tBfQX1OUF7qvMh2XmyHBsRvPOQwnTUnu2yTP0dVL0c++XOgUihm1K+Vqd2bxdAV3WjEkWRzrACwA8FEALwvHdwH4ZqUmVeus9/bnfn3tDvzt5TUA1K4qOcYR9yWKpGimWH2LloLqMnkvcMbU1ctxlcTf/utC/72atKbOOISsQ5i3LH7bFrdqPupGSsrIctNx1RXRgPtZcwtp3KBe2HdY38Q5A0Iaq5A2m1NYHMdNHmrUy0gVF4lal+l9V9x6Gj0wvB9GKfUDtbw6Vs2thqfrU8ufaSnEKg7G2EIAC4nobsaYeQOabg5vabF43U7/mDo4Ho5x+McVYy5aswM7W9r9/SB0okSlfJI8Yfz1ds9tVGCamEzMQFxpAKYxDsdor+gCc4VcX6mdeS7joKVd7+ZSBcdDfZ0oaHbYu9582xmlxeEpSvFYsStJhyhxvxATBvauw3VnH4ojpbRU/nctZp/uWpZxcVZZpZoclkJg+dXyp1o8pt/gGUQ0h4jeJqIVRPQuEa2o6MxqGL6SHSus9uJiHEFuffy43/rLwsR7q1ubBPdesakp8jpfufNNmwqMKVtqm/avMsqqypCh4mBwCNhvRNgiUMVRGnMZnOL1DlIFx8P7cwd7n6dSHIoMqpwTtTiKlQcE4NhJw3DxrP1wxelTEs+P47SpoyLNFIOmm0XMrYsJuTQxjmrRtT5Rc0wVxy0AfgXgKLj7ckz3fvdIuK+/X2MgkMQV/KotTXhg4bqIiyiJNduEeEAaV1XCubLlUyiwiMXhkNo19K2/vBZqj+Luxpf875AhU8XhCqwpI8PtI1StNrIOYZ+hbpuM+pwT+afMKNJVgSDbygSV+ymryKoyFQjyeUQExyH817H7atuSlIJpAaCKribkuoKi6wJTLApTxbGDMfYIY2wjY2wL/6nozGoYnpYqtn0QZe6nbnwe/3PPq77CMN0vQxSWukC16p8lKajtu6q8eecZUxahqaymv7+6FvcuWO0/r8tEYwsqeIwjCeZZHHtJe1er4igFxnzB3pDNRCwOcX8OhwjfPmmSPxdT+KniJf64ouIoUiKIl/H5lzU4HrM/TBLdVchVk54aHOfMJaKrAfwdgL9Vl9d6pMfBLY5H3giqiEXhzbvLbt7tflRy0Fkn50UfvT7GESWp49SNDgAAIABJREFUPbYqOC63O2/PMyxcs0N5/S8ff9t/bOKmAnhWVfK5rquKIllVKlcVQ2DtqNJxZYvjouMn4qLjJxrN1x9DEOb8Yw3qOMKusGIoh7srjmKC41ZhVI7u+tmaWhwz4bqnfgbgl97PL4q9KRF9iogWE1GBiKZLr11KRMuI6C0iOlk4Pss7toyILin23uWAB13f2bjbPybKbu6C2OhtGetnVXlfIp2gT9gdNTSGSFJWDg96B8HxqKuqLV8ItcaQX+OYpOICrsvHIPnKD443SAFjlauqwJivHOqz0ZoNsSFgsSs9vvoPB8eLj3HI54lPK9E5NW5HSh1xWwPUMvuPdONicrPGWqKLfaTGGEkBxthxip/jS7jvGwDOABDaN5WIpgA4E8ABAGYB+B0RZYgoA+B6AKcAmALgLO/cqqBqacEYQ6HAcNn9r/sWx4adbtquXATXrqnMFgWfqAzu+fIR/mOCqwiu+/c7fuwhqRFiXrI48ooYhym6bq0qTC0Oomgdh+o+BSZYHFknFI8Y0CuHc4TuwsUKQVUdBJ+LeKxY95KqbUg5iduRMomu5lY5fvJwzP32sTht6qhqT0VLV1PGppg2ObxcdZwx9qNibsoYW+KNK780G8CfGWOtAN4lomUAZnivLWOMrfCu+7N37pvF3L9UVC05CgxYvml3aB/ujTtdiyMQ7O77bddsiye6WsSwyBhh4yEiwuJ1O/GLx9/Gc8u34O4vH2EU49iwswUPLFznPy82EyXNStbEgmKexdEoKw7FxYePHxiyOMQznr/khJA1VM64QVDHkf7akw8YgSeWBPubVNriOGvGWMx5cwOmjilir4ouKORKaSNiKR7T5WOT8JOHu/IfX4H5jAawWni+xjumOx6BiC4gogVEtGDTJrXrpVRUFsdd81fhS3cuCB3jLh7Z4tA1/tNt2iMHivli/Lnlbn5CUux9T1se7wp7VKuyqkwR244nwd/ngaP1roTF63bAUVoc4RtMGt4HN31ueqA4pI2ciOA1BPSeJ09PCZ/HTKE+ItiiNv2on5o+Fq9fEWxRKyq0SnSwPX7ycKy86lR/J8A0dEG9UbPUcopwOTDdyOmX4nMi+gWAx+KuIaInoN5e9nuMsX8azzAljLGbAdwMANOnT6/In0/XBHDVFnV7DTmrSne9bgUaqoimaHpvkhK4/bmVuP25laHzi9lzHEiXocTfZ6+YdNiXVm4DgEiMQ45hHDi6P/rUZzFjwiCcOnUkBveuwxYpTRhwFU57nhXtIujXkMPj3zwa4wb1wlE/nwtArOMobkzx/ZfD3VUpam0+3YPu+ZmaJ7iH6QVgTNwJjLETixh3LYCxwvMx3jHEHO90eHDclM272/DjB9/04xZtGleVLn1VbM53+T8Xh/b8nrt0Y2rrIa8IjovUZZ3Uyk0Ft8z6GBTgNUiKQs7e4p/NAaP64/qzpwGQhLD3D5rhisN4llHcnRABntvm+IqjyIC78Fj8E9fatqK1NRtLLWPkqiKi14lokfezGMBbAH5dgfk8AOBMIqonogkAJgJ4EcBLACYS0QQiqoMbQH+gAvdXsnFXS0hZtBsW9IncMu9dvLzKXV3rguO61bzswvrZw0v8x+fd/lJq68HtVRXfB0qHPMWxg/Qukc1eVpm805xyXGlgWXGoAu1iMDewOHggu/xisOhqcYWCcx/XFjWmxyw1jGmM4zQAp3s/JwEYxRi7rtibEtHHiWgNgCMBPEREjwEAY2wxgHvhBr0fBXAhYyzPGOsAcBFc99gSAPd651aE7c1t+MpdCzD3LTeoOeOnT+JLdyxIuCoZnt1kEhwXkRWH7BLjW6We98HxZvMosNhMrDjF4bfk8J6fcuBI7bm7vA2rDh47wGheInIdh8oaUxXTZUsIZMtcfPJkAPD7hxVtcVAQewlZHNGmu1Wlq2VVWapHUlv1BgBfBbAvgNcB3OIJ8ZJgjP0DwD80r/0UwE8Vxx8G8HCp9zaBiPDY4g04fPwgHLffMADAs++4XV51LhwjvEV+m8Zi4UJZ3jM7KQV2o5f2O6RPdJMnFao6DhFVjQSHC09+ddzeHLwT7DDF5lNJyBaISqmqCvICBVO6EPz04WPx6cPHJp9oAMH7zEIB/doS1DU2HUsNk+R8vgNAO4BnEdRQfL3Sk6o23Ce/s6Uj0jF20mWPFD0uF9Y65ZN1CCs27cZ1c5eFjicFpH/hVXbnTPJf4QrzOPdWXHW4vOoWrZMTJg/Dk0uD1NOnvn0sdrd2YN32FqN5xaG0OMTH3pNMiqyvtJQSkyBPi6piHNVOwLEKw5KWJFfVFMbYOYyxmwB8EsCHOmFOVSfjEPrWZ7GrpT2xnQc/3wQ+ki44nnEc3P/aOvz9leLi/ibFeXUZB/lCtFeVSJwCkm8hKplrzjwk9Npeg3vjgFH9y7KNaUYxJ0exevdjHOlvmUiK2kctojuoEum4pVBj07HUMEkWh78HB2Oso9ZM60rStyGLXS0doVTauNiESSNDbnHsalFvbZJxgObW4j2BJhZHXdZJTMc1WVn/9StH4r5X1oYUh+66cnxtlBlnikOVtDjKEQMId9itsf+nGpuOpXZJUhwHExHfrYgANHrPCQBjjNVuk5gS6duQw66W9pCy2N2iFupZh9AmHVPv9Ob+3rFHpzgcNLWlS/UNzcOgOVS9rzj0msNkgTB9/CBMHz8Idz6/0j+mW0GXY8GhzKpSDKvbH7wclMNCCAf0vWOlD1sWak6RdWGq7X6sNLGShjGWYYz18376MsaywuNuqzQA1+LYuSdscTS3q4W6ylXVKxcNMCcpjqxD2NNWvMVh0sbctTjiiwbjhpGFiyigdcLaVIjPv/QEXHD03srXVO9NNW62khZHWRSHDY73JLrrZ1oGr233pG9DFrta20PtQX7yoLo1lkqoNdbpM5N2ai0OQnNKi2O8sI+FrmWJSH2Wxzj054hf9rh2IYCZ4jD93xnRvwEj+jUoX1MpZ9W4GT/GUYk6jnK4qmyMo1iuOH0K/nLBEcknWiqOVRwaXFdV2OIQ998QUbmI5N5LQFB0p7M4bn9uJR5/cwP2Gdobt35huvIckZcvOxEfPTjoDGoUHDdwVYnC7eiJQ7HvsD7ac8W3rksSKGZvCPmStBZHJaRgOdxftuVI8XzhgxNCPcQs1cMqDg08OK4LiIuohJqqzUawE1+8B3RE/wYcO2lY8n2l3fhMguP12UxicFwcM+OQcjc+jskKOs3KWneqat9wlZzjyqsSMY5yjBhOx/XGrRF5XSvzsNQ+VnFoaMxl0NKeN8qWUq20VYLOtKdUYy5r9E+cy4T39Ta1OJrb8tiyu1V7jiOtinNSk0WR8K57uuh44rQSx1Dtz60MjkuV7eWkLBYHop9XrXRStXrDYkqxTQ67PdmMg44807ZAD51rqDhM5UNbvmDkNsg6TkjIm8Q4sg7hhXe34dX3tmvPEQVkhuItDpMaljQCVzdcvwaFxaEQdfwzqN3gePC4VmIcNTINSxfCKg4NdRlCW74QaWGuQhXj6K0Ijidt8crZ3iwn96rJZSjUmiNOwHNMhJ94ikPxLjBZwZ33wfHoL1kHqVbqJVsclSsALHdWVc11x62t6VhqGOuq0sCVgUlvKmOLw9Dk2KZQHOceuVfkGBEhI/y369Jx7/vakf7jDTv1LipOKG4hxTjkO2QkafOD0w/AN06cJM0z8ZaR8eVLZGUkz9Ofj5+OW4k6jnJkVQWPraC2dFWs4tDAheUeTe2GiDLGobA4TGMc25uiWVc/nH2g8lxRmOkKAA/ba5D/eLdBZbrcdTZOYBoYOemC41pXlcLiUJxX83UcwmP+uVZbgdRIiKVbUStxq0phFYcG7p4xURyqlX4vhcVhEi8BgGs+c0jySR5hV1WyBNpjUCciKorTpo6MLSw0W4WnSMcVzr37yzP9x31VMY6YrKpK1HGUxeIIJROUPFyZqbkJdXm66ydqFYcG3+IwELQqi0OVjmuSoXXVGQfhxCnDDWboEgqOG2RVNRlUpnOBdvUnp2LsoF5oEK0nSdqZCNM0Fod47gf2GeI/VtXFqNxRFQ2Ol3mMWotxWCym2OC4Bq44TLaJVQnsXgpXlYniSCtMMiktDhMTms+Bj92oENqq+yeNZ4J86j/+6wN4JSYDTKaywfEypONSLVscFosZ1uLQwFeu3OL421ePxFDNhkSqhb4qOG6iONIKEzmQXQ6C3ercByolmOae6YLj4ZMPHTcQ5x81wfj6yu7HUfoYocrxbuvIsHR3rOLQUCcFxzMOxVRGK2IcKovDYLmf1uKQay7KAV8VOyYWh5GrKn1aVbGre1nplZNSLA5+pXVPWboDVnFo4BYHd5PExQ/UPZOi5xu5iWL+IqqVt0mvqLQ4FP4tNmyU72CyuVEx6bjFwhVZzVocpQ9RQbp5KpClbFjFoYHHOP61cB0AVyjrXAuykBrVv6FoIRO3Iv3+aVMU9y6/q8qPcVCyxWGygk7jkuHvp9h3EijPClgcZdnIqfZUR+3NyFLrWMWhQQ40ZzOkXcXKwuDB//lQ0QLC1B1y39c+ACDsKorTG69d/mG8fNmJZnOQ5hIX4zAKjqf4lvHRil37UgUtDirHf4swL2ZX+N2Y7v23rYriIKKriWgpES0ion8Q0QDhtUuJaBkRvUVEJwvHZ3nHlhHRJZWeo9y+w7U41MiyM0OUKFBL7SR72F4D/XmJ99UxoFcdBvepx02fOwzTvWt1yMJXTIWVb2GWjtt5a1r+Z6vEHctTOR4dwwbJuy+13qq+WKplccwBcCBjbCqAtwFcCgBENAXAmQAOADALwO+IKENEGQDXAzgFwBQAZ3nnVgw5RpFzHO2XQBYGjpO84tVVeacVTqEqbwOtc/IBIyItQaJzcH/zmIyq3QfHxOIo5l+naFcVD+zXaFv17ilGLD2NqtRxMMYeF57OB/BJ7/FsAH9mjLUCeJeIlgGY4b22jDG2AgCI6M/eueot+cpAXTb8L55J0egv6ziJArUu4yj7YImX/eHz07GXsMOfCvE+psKyLhu/XuDj8KaMJ+w/HIeOG4BX39ue2KtKRZpVV6kGfiVdVeWpHI8esy4rS1ejFmIcXwTwiPd4NIDVwmtrvGO64xVDtjji2m5E3DdOspDRFeuJQvbEKcMxcXjf2HGKScdNUhx8GF52knEIXz9hYuy58XM0mpY0cBHXQGw5Un7K06uKlI8tlq5ExRQHET1BRG8ofmYL53wPQAeAP5XxvhcQ0QIiWrBp06aix1HGOAzjEhnSn8spl6sqXABodk1ShblvcQgrYZ0FZeSqirQpAWZMGKQ526PIRXglu+OWez8Oa2lYuioVc1UxxmJTeIjoCwBOA3ACCzaqWAtgrHDaGO8YYo7L970ZwM0AMH369KL/MyNZVTGKQ7ZOMg4lrv5z2v25zefI76V6HEd9gsUByeJw51W84vCHpSBuYro3SVr4NGvVVaVSaNbysHQ1qpVVNQvAxQA+yhhrFl56AMCZRFRPRBMATATwIoCXAEwkoglEVAc3gP5AJeeozqrSCHtJeBJRYqC6fBZH+mvrMvr0WnEcUbjrxja5Jx9HPDdRbxTrqvLrQGozOC7+vRqy7t9h5ICGMoxcOt29FbilfFSryeF1AOoBzPFWYPMZY19ljC0monvhBr07AFzIGMsDABFdBOAxABkAtzLGFldygvI2rHG766niH0kCVRczSd2rqiLBcfd3IaQ4+PykpAEDi4OPkiFCHgxEZLw3SVoq26uqDBaHoH7GD+mN3551KI6eOLTkcUuhu6aMWipHtbKq9o157acAfqo4/jCAhys5L5E6SVE4MXELlfBMkqc6gVtSrypDt1EfaW8LIuCmcw5DU1sH9hnaB7f9ZyWA8ApUn4qcYq4OAK/ZsK7fY6kurFIrz+PHLn0M+fM6/eBRpQ9qqTm6u/Vm26prkC2MXEbv/FDFM5KEeLkUh2nluIi8V8jAXnU46YAR/nM+TCGkONRjmbmq3N/iXMX/K9VnUayrid+jEv+35W6rbum+8NqnhlwtJK6WH6s4NIiuqhvPOQxEpP2nV9V4JLqqvGtO3H84LjllMk781dPedenmKcblixVK8iqfj6NyJ0XqOIwm7MU4xHO9sX/5qYPxoUlDVBcVBdf3lXKFFYvjEFBgdg+OHsIVsw/AlFH9cNS+5ftu1xLdUx2WAdHiSKoNUFkciem4nsQf2rce+w7rI1xXvKsqDdedfah/X1nEysWPgN70TpNVJc6VWzP7DuuDYX3LFxzmn1+N6Q3Ue9+nWmxyWKkMt55Mv4YcvvShvbuthWktDg2i4vAD2SliHEkCVRccLyUdNw2nTR2FoX3q8Zmb50eE7MUnT0bWcTD7kKj/vZheVb6rSpgrr2EotyAtV2v5clOfc7CrtbZbjnRTGWepAFZxaEhTH1FMVpU2xpFwr9+ceQiaWoPtbEuRk9xdJq84B/auw48/dqDRGCb3500S9x3aBy82bQUAFLxuK/LHVOraNxPjZqsm9V7qbZpOwRZLrWIVhwGJrirv9ckj+uJjh7qdUJJbjujqOOLnMvuQcKeVUlbsGU+KmYhYnTvDZIU/dlAv3H7e4Zg4vC8+eNW/AQSCXTf9Yt9W0C6l1hQH3wvdLustXR+7/jEgqY0Ff/2YSUPx1WP2AVB8Om5an2gpdQu+pVSCjDXdPOrY/Yahb0N0nVJuQco/jxrTG37tjHUHWboDVnEYkE2wOLIKxVJ8jKO44Di/6j+XHG98re+qSnXHMGn2ORfP5IK93K4bX3GUd9iS8S2OGtYctaZsLbWLVRwGOAmretWq2zzGEf5vTZ2OK+0/MXpAo/G1cR1/ZfgsZQuhmKwqQuBKKndwXNUupRbwYxw1qDdqWZlZahOrOAxIErBZf5UbCKvk7rhlsjj4jndF/O/7MY4ShGya+4rvrbdXhBhRPCXK+0BxlDZOuan3CsHyupJ5i6ULYYPjBiRuA6vwqydXjnOdHT4vrQIopakfV3ipZJl0m1SuKuHUG86Zhn++tg57D+ltchtjVH22agHuqlJt3mWxdDWsxWFAkFWlCY7zFFBBAidmVWldVWktjhKC45mopaTjkLEDMGPCIPzg9PCOvcUWAI7s34ivHrNPxE1S6h4VfLga0xu+q6q1BhXH8ZOHAQBG9K+NLr2W2sdaHAZkE4RzRrFyT1IA9Tl1a/Oig+NFuarML2rIZXDvV46MHE/jH+8MV7pfOV75W6WCZ1XVouL42jH74MzDx2Jwn/pqT8XSRbAWhwGZhNQfVW+nJJncq06nOFLOTQqOA8DVn5yK+y/8YOK1WT/Gke6excLn+u2T99OeU2p6rqP4W9QCteyqchyySsOSCmtxGBDnxz9j2milsE+yHHSKI22GCz9dvN+npo/VnB0mqBxPdcuicRzCyqtOjT2nZFcVH6e29IavOFo78glnWiy1j7U4DODdb1VC/VefPkS5yk0qjGvQuqpSzi2hxiQOVTZYrVBq5XitpeMe5W3WNGVkvyrPxGIpHWtxGBBkLqlRtbko3lXVeTGOWuzUWipOjcY4PjxlOF79/ocxsHddtadisZSMtTgMkNt6yEFiVSvvpMCzbvvW4us40isB3i+LZ9V0BwIlXt15qLBKw9JdsBaHAXJWlUPufhbcauAiO01WlS4InHrPcT84nu46wFVu875zHIZ0o8BooMRrUHNYLN0EqzgMkOMVjkM4bWqwV0WgJMzrOEzvZUqxbSPGDOxV1HWVolR57/dttHrDYqkY1lVlQFYqAJSVgl+tXIgeS0IWcGn1Br++u0Urik3L5dfVYsDfYukuWIvDgKTW5ergeJEWR5Ft1ccMNG9uWG5+e9ahmKBpHVJpZh0wIhQ7qNXKcYulO1EVxUFEPwYwG0ABwEYAX2CMrSPX3/IbAB8B0Owdf8W75lwAl3lD/IQxdkdnzVdOeZX956pq5eJdTunOH9S7Dr8961Acuc/gou5XDk4/OLrFbLGklfc3fu6w0PNa7VVlsXQnquWqupoxNpUxdgiABwFc7h0/BcBE7+cCADcAABENAvADADMBzADwAyIa2FmT9QsANamexVQra1u0F2GpnH7wqG4V4C6NIho3WiyWVFRFcTDGdgpPeyOQxbMB3Mlc5gMYQEQjAZwMYA5jbCtjbBuAOQBmddZ8nYQiu7hq5aF9owL9yL311kE5ayt612UwY/ygso3XFbCuKoul8lQtxkFEPwXweQA7ABznHR4NYLVw2hrvmO64atwL4ForGDduXElzHNq3Hpt2tUaORwLaDj8efuHWL0zH/iP74cgr/x06fs8FR+CeF99T3rOcG/0s/lGn6dayU6z+VGW4WSyW8lIxi4OIniCiNxQ/swGAMfY9xthYAH8CcFG57ssYu5kxNp0xNn3o0KEljfWvi47C7ecd7j8PhJkU49C4R46fPBwj+6cLWvf03dgOGt0fAHD0pOL+dqqaGovFUl4qZnEwxk40PPVPAB6GG8NYC0Ds0DfGO7YWwLHS8adKnmQCI/o3hPYo0LmkfPdIGe5Zi1uLdiYHju6PN390MnrVFffV1Fl/FoulfFQlxkFEE4WnswEs9R4/AODz5HIEgB2MsfUAHgNwEhEN9ILiJ3nHOhXdXg/FBMeP3c9dUX925l7KsXoyxSoNQGxxX67ZWCwWmWrFOK4iov3gpuOuAvBV7/jDcFNxl8FNxz0PABhjW70U3pe8837EGNvauVNObnKYZpU7sn+jssW4VRylYSMcFkvlqYriYIx9QnOcAbhQ89qtAG6t5LxMiVZ7l29fC6s3SsP2qrJYKo9tOZKCs2e6WVpylXQQkC1dWFmLozRsryqLpfLYliMpOGPaGJwxbUzk+CHjBgAAzjy8tPRfwAbHSyXIcLOaw2KpFFZxlAFdvILz5Q9NwLRxA/G1P72SOJa1OEqj0hbHNZ85uKTgvcXSHbD/AZ3A906dYnyu1RslUuFeVR8/NGpxWiw9DRvjqDF6egFgqQzs5XbK3Xtodbr1Wiw9AWtxWLoV+4/shzu/OAMzJvSsHl0WS2diFYel21FsuxKLxWKGdVVZLBaLJRVWcVgsFoslFdZV1Ync9LnDMEjY5tRisVi6IlZxdCInHzCi2lOwWCyWkrGuKovFYrGkwlocNcIDF30QC1dvr/Y0LBaLJRGrOGqEqWMGYOqYAdWehsVisSRiXVUWi8ViSYVVHBaLxWJJhVUcFovFYkmFVRwWi8ViSYVVHBaLxWJJhVUcFovFYkmFVRwWi8ViSYVVHBaLxWJJBbFKbc5cAxDRJgCrShhiCIDNZZpOLdCd3k93ei+AfT+1Tk97P3sxxrQb23RrxVEqRLSAMTa92vMoF93p/XSn9wLY91Pr2PcTxrqqLBaLxZIKqzgsFovFkgqrOOK5udoTKDPd6f10p/cC2PdT69j3I2BjHBaLxWJJhbU4LBaLxZIKqzgsFovFkgqrOBQQ0SwieouIlhHRJdWejwlEdCsRbSSiN4Rjg4hoDhG94/0e6B0nIrrWe3+LiGha9WauhojGEtFcInqTiBYT0de9413yPRFRAxG9SEQLvffzQ+/4BCJ6wZv3X4iozjte7z1f5r0+vprzV0FEGSJ6lYge9J535feykoheJ6LXiGiBd6xLftcAgIgGENHfiGgpES0hoiPL+X6s4pAgogyA6wGcAmAKgLOIaEp1Z2XE7QBmSccuAfAkY2wigCe954D73iZ6PxcAuKGT5piGDgD/yxibAuAIABd6f4eu+p5aARzPGDsYwCEAZhHREQB+DuAaxti+ALYBON87/3wA27zj13jn1RpfB7BEeN6V3wsAHMcYO0Sob+iq3zUA+A2ARxljkwEcDPfvVL73wxizP8IPgCMBPCY8vxTApdWel+HcxwN4Q3j+FoCR3uORAN7yHt8E4CzVebX6A+CfAD7cHd4TgF4AXgEwE271btY77n/3ADwG4EjvcdY7j6o9d+E9jPGEz/EAHgRAXfW9ePNaCWCIdKxLftcA9AfwrvwZl/P9WIsjymgAq4Xna7xjXZHhjLH13uP3AQz3Hnep9+i5Ng4F8AK68HvyXDuvAdgIYA6A5QC2M8Y6vFPEOfvvx3t9B4DBnTvjWH4N4GIABe/5YHTd9wIADMDjRPQyEV3gHeuq37UJADYBuM1zJf6BiHqjjO/HKo4eAnOXEl0u95qI+gC4D8A3GGM7xde62ntijOUZY4fAXa3PADC5ylMqCiI6DcBGxtjL1Z5LGTmKMTYNrtvmQiI6Wnyxi33XsgCmAbiB/f/27i9EqjKM4/j3V/bHsvxTXgRrhBQJ4aZRECRhFEVKESVYCXpRF1511T8TTLsoaKEogm4qIpIVklykIKJWytLSwHXzD5jBUkolhgqWRNjTxfuMjrOztUfHxsnfB4Zz5j1nZ95HZn3mfc/Z542YCfzG8Wkp4NTjceIYbi8wpe55V7Z1ol8kXQGQ233Z3hExSjqPkjRWRsT72dzRMQFExEFgHWU6Z4KkMXmovs/H4snj44Ff/+OujuQW4F5JQ8AqynTVK3RmLABExN7c7gPWUBJ7p37W9gB7IuLrfL6akkhaFo8Tx3CbgWvyDpHzgQeBtW3u08laCyzK/UWU6wS19oV5N8XNwKG6IewZQZKAN4GdEfFS3aGOjEnSZEkTcn8s5XrNTkoCmZenNcZTi3Me0J/fEtsuIpZERFdEXEX5/eiPiAV0YCwAki6WdEltH7gT2EaHftYi4mfgR0nXZtPtwA5aGU+7L+SciQ9gDrCLMge9tN39GWWfe4GfgD8p3zgeocwjfwp8B3wCTMpzRblz7HvgW+DGdve/STyzKEPpQWAgH3M6NSagG9iS8WwDlmX7VGATsBt4D7gg2y/M57vz+NR2xzBCXLOBDzo5luz31nxsr/3Od+pnLfs4A/gmP299wMRWxuOSI2ZmVomnqszMrBInDjMzq8SJw8zMKnHiMDOzSpw4zMysEicOswaSjmaV1NrjHyskS1ozNhYYAAACR0lEQVQsaWEL3ndI0uW5v+EkX+MFSbdJuk/SklPtk1kzvh3XrIGkwxExrg3vO0S5h37/KbxGPzAXeB5YHRFftqh7Zsd4xGE2SjkieDHXbdgk6epsXy7p8dx/TGUNkUFJq7JtkqS+bPtKUne2XybpY5X1Od6g/CFW7b0O51aSeiRty/edP0LfeiQNAjcBG4FHgdclLTuN/yR2lnLiMBtubMNUVf1/1ociYjrwGqVCbKOngZkR0Q0szrYVwJZsewZ4J9ufBb6IiOso9ZGubPJ691P+Cvh64A6gp1ZvqF5EPEGpFvA2JXkMRkR3RDxXJXCz0Rjz76eYnXWORKli20xv3fblJscHgZWS+iilHqCUT3kAICL6c6RxKXArJTEQER9KOtDk9WYBvRFxlFKk7jNKYmhWP+0GStmMaZy4wJJZSzlxmFUTI+zXzKUkhHuApZKmn+4OSZpBGWl0URZJuqg0a4CygNKR090HO7t4qsqsmvl12431BySdA0yJiHXAU5Ty4eOA9cCCPGc2sD/K2iKfAw9n+92UQnSN1gPzcxGoyZSktKn+hIgYyBHSLspyx/3AXVGWQXXSsJbziMNsuLH5bb3mo4io3ZI7MS9C/wE81PBz5wLvShpPudD9akQclLQceCt/7neOl7ZeAfRK2g5sAH5o0pc1lHU7tlJGOE9GKZt9gkwqByLiL0nTImJH9bDNRse345qNUitulzX7P/BUlZmZVeIRh5mZVeIRh5mZVeLEYWZmlThxmJlZJU4cZmZWiROHmZlV8jd/dyLdaZUe/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): numero maximo de episodios de entrenamiento (n_episodios)\n",
    "        max_t (int): numero maximo de pasos por episodio (n_entrenamiento)\n",
    "        eps_start (float): valor inicial de epsilon\n",
    "        eps_end (float): valor final de epsilon\n",
    "        eps_decay (float): factor de multiplicacion (por episodio) de epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # puntuaciones de cada episodio\n",
    "    scores_window = deque(maxlen=100)  # puntuaciones de los ultimos 100 episodios\n",
    "    eps = eps_start                    # inicializar epsilon\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            \n",
    "            # elegir accion At con politica e-greedy\n",
    "            action = agent.act(state, eps)\n",
    "            \n",
    "            # aplicar At y obtener Rt+1, St+1\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            # almacenar <St, At, Rt+1, St+1>\n",
    "            agent.memory.add(state, action, reward, next_state, done)\n",
    "            \n",
    "            # train & update\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            \n",
    "            # avanzar estado\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            \n",
    "            if done:\n",
    "                break \n",
    "\n",
    "        scores_window.append(score)       # guardar ultima puntuacion\n",
    "        scores.append(score)              # guardar ultima puntuacion\n",
    "        eps = max(eps_end, eps_decay*eps) # reducir epsilon\n",
    "        \n",
    "        print('\\rEpisodio {}\\tPuntuacion media (ultimos {:d}): {:.2f}'.format(i_episode, 100, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisodio {}\\tPuntuacion media ({:d} anteriores): {:.2f}'.format(i_episode, 100, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=200.0:\n",
    "            print('\\nProblema resuelto en {:d} episodios!\\tPuntuacion media (ultimos {:d}): {:.2f}'.format(i_episode-100, 100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth') # guardar pesos de agente entrenado\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = dqn()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Puntuacion')\n",
    "plt.xlabel('Episodio #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizar agente entrenado\n",
    "\n",
    "Podemos visualizar el comportamiento del agente entrenado cargando los pesos del fichero donde se han almacenado. (Solo en entorno local, no en Google Collaboratory.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cargar pesos del fichero `checkpoint.pth`\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))\n",
    "\n",
    "for i in range(3):\n",
    "    state = env.reset()\n",
    "    for j in range(200):\n",
    "        action = agent.act(state)\n",
    "        if render: env.render()\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            break \n",
    "            \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### EJERCICIO\n",
    "\n",
    "¿Podrías extender este código para otro de los entornos de OpenAI Gym? Por ejemplo, [CartPole-v0](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py).\n",
    "\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}