Windy_Gridworld.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Windy Gridworld.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hayatrajani/reinforcement-learning/blob/main/Windy_Gridworld.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iUUXSgXJhhax"
      },
      "source": [
        "# Import required Modules"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BdblfD3eoFIr"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.patches import Rectangle, Circle, Polygon\n",
        "from tqdm import tqdm\n",
        "\n",
        "%matplotlib inline\n",
        "np.set_printoptions(linewidth=np.inf)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IF3KQO08ho4Z"
      },
      "source": [
        "# Design the Grid World\n",
        "\n",
        "![windy_gridworld](https://drive.google.com/uc?id=10CcBBLWgMateeYRp3JFwTI--T1UgS8TQ)\n",
        "\n",
        "The agent starts in the cell marked $S$.\n",
        "\n",
        "When it reaches the cell marked $G$, it will receive a reward of $100$ and the episode ends.\n",
        "\n",
        "Every other step will be rewarded with $-1$.\n",
        "\n",
        "However, there is a crosswind running upward through the middle of the grid causing the resultant next states to shift upward by a \"wind\", the strength of which varies from column to column. The strength of the wind is given below each column, in number of cells shifted upward.\n",
        "\n",
        "The agent has eight (non-deterministic) actions:\n",
        "\n",
        "* It can move to an adjacent cell according to eight-neighborhood.\n",
        "* With probability 0.8, the desired action is carried out. With probability 0.1, the agent deviates from the desired direction by one cell to the left or to the right (axis-parallel moves become diagonal moves and vice versa).\n",
        "* Actions that would move the agent off the grid are handled by truncating the resulting cell coordinates to valid grid coordinates. Hence, almost all diagonal moves outside the grid (except for moves into cells extending corners) will result in a horizontal or vertical move."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7bW9Hi4oQCdC"
      },
      "source": [
        "gridSize = (7,10)\n",
        "\n",
        "startCell = (3,0)\n",
        "goalCell = (3,7)\n",
        "\n",
        "goalReward = 100\n",
        "rewards = np.zeros(gridSize)-1\n",
        "rewards[goalCell] = goalReward\n",
        "\n",
        "winds = (0,0,0,-1,-1,-1,-2,-2,-1,0)\n",
        "winds_ = np.empty(gridSize, dtype=int)\n",
        "for cell in np.ndindex(gridSize):\n",
        "    winds_[cell] = winds[cell[1]]+np.random.choice((-1,0,1))\n",
        "getWind = lambda state, stochastic: (winds_[state],0) if stochastic else (winds[state[1]],0)\n",
        "\n",
        "# possible actions: NW, N, NE, E, SE, S, SW, W\n",
        "actions = ((-1,-1), (-1,0), (-1,1), (0,1), (1,1), (1,0), (1,-1), (0,-1))\n",
        "nActions = len(actions)\n",
        "\n",
        "initPolicy = lambda: np.random.choice(range(nActions))\n",
        "\n",
        "\n",
        "def step(state, actionID, deterministic=True, stochasticWind=False):\n",
        "    action = actions[actionID]\n",
        "    # desired action is performed with probability 0.8\n",
        "    # else action deviates to the left or right\n",
        "    if not deterministic:\n",
        "        rand = np.random.random()\n",
        "        if rand > 0.8:\n",
        "            # action deviates left\n",
        "            if rand > 0.9:\n",
        "                thetaDev = -np.pi/4\n",
        "            # action deviates right\n",
        "            else:\n",
        "                thetaDev = np.pi/4\n",
        "            rotMat = np.array([[np.cos(thetaDev),-np.sin(thetaDev)],\n",
        "                            [np.sin(thetaDev),np.cos(thetaDev)]])\n",
        "            # deviated action\n",
        "            action = np.round(rotMat @ action).astype(int)\n",
        "    # check wind strength (blowing upwards and sometimes varying by 1)\n",
        "    wind = getWind(state, stochasticWind)\n",
        "    # determine the new state and reward\n",
        "    state = tuple(s+a+w for s,a,w in zip(state,action,wind))\n",
        "    # truncate state to valid cell coordinates if agent moves off grid\n",
        "    state = tuple(np.clip(state,0,(gridSize[0]-1,gridSize[1]-1)))\n",
        "    # end episode if terminal state\n",
        "    done = rewards[state] == goalReward\n",
        "    return (state, rewards[state], done)\n",
        "\n",
        "\n",
        "# Online Tabular TD-lambda\n",
        "def policyEvaluation(policy, V=np.zeros(gridSize),\n",
        "                     lmbda=0.1, gamma=0.9, alpha=0.2, decay=True,\n",
        "                     episodes=1000, deterministic=True, stochasticWind=False):\n",
        "    eligibility = np.zeros(gridSize)\n",
        "    if decay:\n",
        "        nVisit = np.zeros(gridSize, dtype=int)\n",
        "    # repeat for each episode\n",
        "    for episode in tqdm(range(episodes)):\n",
        "        # starting from an initial state\n",
        "        state = startCell\n",
        "        # repeat for each step of the episode\n",
        "        while True:\n",
        "            # get an action for the current state\n",
        "            actionID = policy()\n",
        "            # take action and observe next state and reward\n",
        "            state_, reward, done = step(state, actionID)\n",
        "            # update eligibility traces and state values\n",
        "            if decay:\n",
        "                alpha = 1/(nVisit[state]+1)\n",
        "            tdError = reward + gamma*V[state_] - V[state]\n",
        "            eligibility[state] += 1\n",
        "            for s in np.ndindex(gridSize):\n",
        "                V[s] += alpha*eligibility[s]*tdError\n",
        "                eligibility[s] *= gamma*lmbda\n",
        "            # move on to the next state\n",
        "            state = state_\n",
        "            if decay:\n",
        "                nVisit[state] += 1\n",
        "            # until terminal state reached\n",
        "            if done:\n",
        "                break\n",
        "    return V\n",
        "\n",
        "\n",
        "def epsGreedy(Qstate, epsilon=0.1):\n",
        "    return np.random.choice(range(nActions)) if(np.random.random()<epsilon) else np.argmax(Qstate)\n",
        "\n",
        "\n",
        "# SARSA-lambda\n",
        "def policyImprovement(Q=np.zeros((*gridSize,nActions)), replaceTrace=False,\n",
        "                      lmbda=0.1, epsilon=0.1, gamma=0.9, alpha=0.2, decay=True,\n",
        "                      episodes=10000, deterministic=True, stochasticWind=False):\n",
        "    eligibility = np.zeros(Q.shape)\n",
        "    if decay:\n",
        "        nQVisit = np.zeros(Q.shape, dtype=int)\n",
        "    # repeat for each episode\n",
        "    for episode in tqdm(range(episodes)):\n",
        "        # starting from an initial state\n",
        "        state = startCell\n",
        "        # get an action for the initial state using a policy derived from Q\n",
        "        actionID = epsGreedy(Q[state], epsilon)\n",
        "        # repeat for each step of the episode\n",
        "        while True:\n",
        "            # take action and observe next state and reward\n",
        "            state_, reward, done = step(state, actionID, deterministic)\n",
        "            # get an action for the new state using a policy derived from Q\n",
        "            actionID_ = epsGreedy(Q[state], epsilon)\n",
        "            if decay:\n",
        "                nQVisit[state][actionID] += 1\n",
        "                alpha = 1/nQVisit[state][actionID]\n",
        "            tdError = reward + gamma*Q[state_][actionID_] - Q[state][actionID]\n",
        "            if replaceTrace:\n",
        "                eligibility[state] = np.eye(nActions)[actionID]\n",
        "            else:\n",
        "                eligibility[state][actionID] += 1\n",
        "                eligibility[state] *= gamma*lmbda\n",
        "            # for each state-action pair\n",
        "            for s in np.ndindex(gridSize):\n",
        "                # update action value\n",
        "                Q[s] += alpha*eligibility[s]*tdError\n",
        "                # update eligibility trace\n",
        "                if s != state:\n",
        "                    eligibility[s] *= gamma*lmbda\n",
        "            # move on to the next state-action pair\n",
        "            state = state_\n",
        "            actionID = actionID_\n",
        "            # until terminal state reached\n",
        "            if done:\n",
        "                break\n",
        "    return Q\n",
        "\n",
        "\n",
        "def plotPolicy(policy, deterministic=True, stochasticWind=False):\n",
        "    fig, ax = plt.subplots(1,1,figsize=(gridSize[1],gridSize[0]))\n",
        "    ax.axis([0,gridSize[1],0,gridSize[0]])\n",
        "    ax.set_xticks(range(gridSize[1]))\n",
        "    ax.set_yticks(range(gridSize[0]))\n",
        "    ax.grid(True, color='k', linewidth=1.5)\n",
        "    ax.set_aspect(1)\n",
        "    ax.invert_yaxis()\n",
        "    ax.set_yticklabels([])\n",
        "    ax.yaxis.set_ticks_position('none')\n",
        "    ax.set_xticklabels([])\n",
        "    ax.xaxis.set_ticks_position('none')\n",
        "    ax.add_patch(Rectangle((startCell[1],startCell[0]),1,1,color='dodgerblue'))\n",
        "    ax.add_patch(Rectangle((goalCell[1],goalCell[0]),1,1,color='springgreen'))\n",
        "    for i, w in enumerate(winds):\n",
        "        ax.text(i+0.5, gridSize[0]+0.5, -w, fontsize=20, color='slategrey')\n",
        "    state = startCell\n",
        "    states = [state]\n",
        "    while True:\n",
        "        actionID = np.argmax(policy[state])\n",
        "        nextState, _, done = step(state, actionID, deterministic, stochasticWind)\n",
        "        states.append(nextState)\n",
        "        state = nextState\n",
        "        if done:\n",
        "            break\n",
        "    states = [(x+0.5,y+0.5) for y,x in states]\n",
        "    ax.add_patch(Circle(states[0], radius=0.1, fill=True, color='darkviolet'))\n",
        "    ax.add_patch(Polygon(states[:-1], closed=None, fill=None, edgecolor='darkviolet', linewidth=2))\n",
        "    ax.arrow(states[-2][0],states[-2][1], states[-1][0]-states[-2][0],states[-1][1]-states[-2][1],\n",
        "             head_width=0.3, head_length=0.15, length_includes_head=True,\n",
        "             fill=True, color='darkviolet', linewidth=2)\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "def plotStateValues(V):\n",
        "    V = V.copy()\n",
        "    V = np.round(V,2)\n",
        "    fig, ax = plt.subplots(1,1,figsize=(gridSize[1],gridSize[0]))\n",
        "    ax.set_xticks(range(gridSize[1]))\n",
        "    ax.set_xticks(np.arange(-.5, gridSize[1], 1), minor=True);\n",
        "    ax.set_xticklabels(range(1, gridSize[1]+1))\n",
        "    ax.set_yticks(range(gridSize[0]))\n",
        "    ax.set_yticks(np.arange(-.5, gridSize[0], 1), minor=True);\n",
        "    ax.set_yticklabels(range(1, gridSize[0]+1))\n",
        "    ax.grid(which='minor', color='k', linewidth=2)\n",
        "    ax.invert_yaxis()\n",
        "    ax.set_title('State Values for each Cell', fontsize=18)\n",
        "    im = ax.imshow(V, cmap='hot_r')\n",
        "    cbar = fig.colorbar(im, orientation='horizontal', pad=0.1)\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSzKvvzD7o4h"
      },
      "source": [
        "# Task 1\n",
        "\n",
        "Randomly generate 1000 episodes according to the initial policy.\n",
        "\n",
        "Compute the state value V(s) for each visited state using TD(λ) Policy Evaluation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CvbRcWIp4AkD",
        "outputId": "7b14cbc1-a99d-49fd-ada8-8a4fee3fc4a4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 645
        }
      },
      "source": [
        "V = policyEvaluation(initPolicy)\n",
        "print()\n",
        "plotStateValues(V)\n",
        "print()\n",
        "print('\\nState Values for initial policy:\\n')\n",
        "print(np.round(V,2),'\\n')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAGSCAYAAADn8RuZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de/xsdV3v8ddbNiggigoSiYUmkeQp1B1HU0lADC/b2+kUVgRmYqYmnYrIOg/p9KjsYmnHssc+QmDeMhRvGYIKoaXk3ojKxXuoIIKIyk3l4uf8sdav/WP2zG/P77L3mvnu1/PxmMfMXrN+a94za34z79/6fmd2qgpJkqSW3W3oAJIkSdubhUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPFKjklyQ5Mqhc0wjyR5J/jrJl5LcOS+5VyvJqUkqyYFDZ5kkyRlJamTZzOeWRll4NFeSPDjJxiSfSnJrkm8kuSLJmUmOGFn31CTPWIPbPCnJCavdzpjtPqV/03jlNtb7g369n1/rDDPkd4AXA/8InACcNGiaOdSXxpOSfDDJDUluT3JtkvckOSHJuqEzSkPyF0BzI8l64F+B24HXAZcBuwMHAU8EbgLOX/QjLwPOBN6+yps+CbgSOGOV2xl1DvAV4BeS/HZV3T66QpIAxwPfBN62xrc/S44GPllVvz10kHmU5CHAPwM/DLwP+BPgeuD+wBOAvwcOAU4eKqM0NAuP5snLgD2AQ6vq46NXJvm+HR9p5arqziRnAC8FNjC+0BwJ/CDwt1X1nR0Yb0f7PuBLa73RJHtV1U1rvd1ZkmR34N3Ag4H/UVWjz6M/TfITwE/s8HDSDHFIS/PkIODr48oOQFV9FSDJgYvmHBzfDwfV4nkISX4uyTv7OSPfTXJ9krcn+bHF2+x/5geBn1q8ncVzF5KsT3J2v43vJvl0kt+bcgjhdKCAX55w/S8vWm/q3JMkuTLJBWOWP76/XyeMLL97kpcmuSzJd5J8M8m7kjx8ZL279cMpn0hyU5Ib+8fhtCS7LpHnhP4xfhB3fYxPXbTOM5L8W5JbktzcX376pPuW5OFJ3pvkW8AnpnhM9k/ymv4xvS3JV/ph0/uPrPf9SV6R5JJ+KPU7SS5P8jtJdhmz3d2SnNyvf2uSbyXZlORFY2LcPckfJ7mq368fT/LkbWXv/QpwMPCKMWUHgKr6aFX97Ui+1TxvpbnjE1vz5PPAwUmeNemFvfc14DjgH4APAhvHrPMi4Ov9dV8Ffgg4Efi3JI+oqs/26x0H/BXd8MAfjdwGSZ5Cd2Tmc8ArgBuARwP/BzgU+J9L3aGq+nySC4FjkuxfVdcsXJfk3sAzgY9X1eZl5l61vqicA/wk3WP5auDewPP62zu8qjb1q/8e3X1+F/B3wJ10JeZpwN3phiHHuZDxj/En+gy/BvwN8Kl++9DN8Xl7kudX1ei+/QHgA8A/AW8F7rmN+/gDwIeB3YDT6J5jDwFeAByRZH1Vfatf/ceAZwFn9+vtChwDvJzu6MrzF213N+C9wOOBc4HXA98B/lu/jVePRDmT7jH6iz7LSf19/OGqunKp+wD8TH8+7nk+6X6v6nkrzaWq8uRpLk50L8i30R0R+QzdUY8XAA+dsH4BZ0y4bs8xyx4KfJdu+Gjx8iuBC8asfw+60nEhsG7kut/ob//xU9yv4/p1Tx5Z/vx++a+vMPcFwJVT3pfH97d1wpj78NMj696LbvjpgkXLLgYuX8W+3SoXcB/gZro35XuN3P7n6eZs7T2yjQJ+ZRm3+w7gOuCAkeXrgTuAUxct2x3ImG38A13B23/RspP7LH88Zv27Lbp8ar/euxdvm274qYA/meI+fB341jLu87Ket3Rz12pkvYXcB650n3vytKNPDmlpblTVh4FH0v01fG/gOcDfApcnuTDJg5exrVugmxSc5F5J9qE7avNp4L9PuZmjgf3oJoTunWSfhRPwnn6dJ06xnbOAb/X3Z7Hn0BWZ169x7mn9It2Rlc0j92034Dzgsenmj9Dnf0CSx67h7R8N7An8dVXduLCwv/zXdEdvnjDyMzfQ7Y9t6o+gPRV4J/Cdkft4JV3R+q/9V1Xfrqrqf3a3JPft130v3fSA9Ys2/wvAN9hyVIpF2/nemDivWth2v85H6creQVPclXvRlb9prdXzVporDmlprlTVJ+mGNEjyg8BP0c1heBzwjiSPrKrbtrWdfg7KH9Id2dhz5Or/nDLOQ/vz05dYZ79tbaSqvp3kTcCvJnl0VX04ySF0BeYtVXXDGuee1kPpjmp8bYl19gG+TDfx+u3AB5N8he7o0j8DZ02zPyZ4UH9+2ZjrFpaNltzPV9WdU27/YLqi8tz+NM4XFi70c1tOAX6JbtgrI+veZ9Hlg4BLavqJ5l8Ys+zrwP2m+Nkbgb2mvB1Yo+etNG8sPJpbVfVF4HVJFubqPAY4DPjQUj/Xz9u4kO6N4g/pjo7cQneI/pVsY97H4k31578NXDJhna9Mua3TgV+lO6rzYbYc7TltjXPXhOXjXgsCfBL4X0ts72vQHX1L8kPATwNH9KefB34/yWMXl7bt7NZlrLuw/15Pd9RwnG8vuvyXbPmuoD+iGwq7HXgE8Kes7kMgk0raaKka51Lg8CQPrqpxxWnSNtfieSvNDQuP5l5VVZKL6ArPA6b4kWfSlYOnVdXi7+0hyf3ohpHuchMTtrMwQfiWqnrfMiJvpao+muSTwM8l+S26eT1fovtOlZXmHucG4L5jlo8bDvwssC/wgQnDMKP34Wa6icJv7TMtTDh+LvDnU2QbtfDm/aPA+0euO2RknZX4HN2+3W3K/XcccGFVHbt4YbrvwBn1GeBHkty9qqbZL6vxVuBwuiOdL51i/TV73krzxDk8mhtJjh73kdl+HsnCnIPLF111M+Pf3Bf+mr7LX89Jnkf3fTCjJm3nvXR/5Z+SZKvrk+yeZDlDDafRzcd4Ld2QwhkjRWO5ucdZeCP+r2KY5O7AC8es+7p+u2OP8CTZb9HlfcascnF/Pu6xm8Z5dEewXrz4cewvv5huv5y3wm1TVV+nm7PyrCSPGr2+nye176JFd7L1Y78n3UTfUW+gG+L6/XHbXWnmCV5Ld7TvtzLm4/r9bT6yL6Cw9s9baS54hEfz5K+A+yV5J91Qy63AA+mGTn4YeF0/x2fBR4AnJPkduqMlVVVvBv6l/9l/SPJqusmljwGeTPfpn9Hfi48Az03yh8AVwPeAd1XVLUl+iW7uyqeTnE531GBv4EfoPn78TLr5LNN4PfBndB8JLraefLvc3OO8GjgWeF+Sv6ObgHwc44eCXkU3wfXPkxxJ93HvG+k++n0U3cesF/47jyuSfAS4iG44ZH+6j8vfBrx5ilxbqapvJjmZ7ijRRem+pBG6OVwPAZ5fWz4yvlIvoBsCvTDJ64CP0f0h+GDg6XSl79R+3bOA5yf5R7ojb/vRfU/S18ds91V0Xyb5++m+9O9cusfrR+nmDo1Otl6xqro1yVPp5ky9Pcm5dEXw63RH6I6gG2r8s379tX7eSvNh6I+JefI07YnuKM7fAB+n+86WO+he1M+ne+O528j6B9G90dxIVyBq0XWH073R3UT33zb8M/Awxn+U+/50wwY30JWdu3wct/+51wNX073BXwv8O/C/gfsu8z7+Y7/990+4fjm5t1rWLz+e7ojAbXQTnU+m+0bnu3wsvV93HfDrwEfpjrbcQjck8gbgiYvWO4VuftF1dENrX6b7LpxHTHm/r2TMx+X7657ZP54Lt//vwDOWs41t3PY+dENun6ErJd+kK9SvAg5ZtN4e/Xpf7Nf7bH+/j5rw2N2D7vuJLlu03Y8Cv7ZonVNHn08rvT99vt/onx/foJtfdG3/HDkO2GVk/amet/ixdE+NnFI1aXqCJElSG5zDI0mSmmfhkSRJzbPwSJKk5ll4JElS8/xY+iol2UD38dPnDZ1FktSk66tq322vpqX4Ka01kqSqvr3tFQey8H88Vt0+cJLJkl2Beck4m783C99pN6v5wIxrZb4yzuZr45bXxaX+u7hh9d99ubmq1m9rXS3NIS1JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRaeCZKcnuS6JJcOnUWSJK2OhWeyM4Bjhg4hSZJWz8IzQVVdCNwwdA5JkrR664YOMM+SnAicOHQOSZK0NAvPKlTVRmAjQJIaOI4kSZrAIS1JktQ8C48kSWqehWeCJG8CPgwcnOSqJM8dOpMkSVoZ5/BMUFXPHjqDJElaGx7hkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8/7f0NZTsPnSEbUp2HTrCNs1HxgwdYUmzng/MuFbmI+NsvzYm+w4dQTuAhWeVkmwANgydQ5IkTZaqGjpDE5JU1beHjjHRwl9Y85Hx9oGTTLZw9GlWf28W/tqf1XxgxrUyXxlvGjjJeMleAFR9cOAkkyWPA9hcVeuHzjLvnMMjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FZ4IkD0xyfpLLk1yW5CVDZ5IkSSuzbugAM+wO4Der6uIkewGbk5xXVZcPHUySJC2PR3gmqKprquri/vJNwBXAA4ZNJUmSVsLCM4UkBwIPBy4aNokkSVoJh7S2Ick9gbcCJ1XVjSPXnQicOEgwSZI0NQvPEpLsSld23lBVbxu9vqo2Ahv7dWsHx5MkSVNySGuCJAFOA66oqr8cOo8kSVo5C89kjwGOA45Mckl/evLQoSRJ0vI5pDVBVX0IyNA5JEnS6nmER5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5qaqhMzQhiQ+kJGl72FxV64cOMe/WDR1g3iXZAGwYOockSZrMIzxrJElV3TR0jImSvQCYj4zfHjjJZMnuAFTdPnCS8ZJdAZjl3+skgBlXa0vG2XwuwuLn42cHTjJechAAVT85cJLJkn8Hj/CsCefwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZkgyT2S/EeSjye5LMkfDJ1JkiStzLqhA8yw7wJHVtXNSXYFPpTkX6rqI0MHkyRJy2PhmaCqCri5/+eu/amGSyRJklbKIa0lJNklySXAdcB5VXXRyPUnJtmUZNMwCSVJ0jQsPEuoqjur6lDgAOCwJA8buX5jVa2vqvXDJJQkSdOw8Eyhqr4JnA8cM3QWSZK0fBaeCZLsm2Tv/vLuwNHAp4ZNJUmSVsJJy5PtD5yZZBe6YviWqnr3wJkkSdIKWHgmqKpPAA8fOockSVo9h7QkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmpeqGjpDE5L4QEqStofNVbV+6BDzbt3QAeZdkg3AhqFzSJKkyTzCs0aSVNU3ho4xUXIfAOYj400DJ5ks2QuAqtsHTjJesiswu/lgccbZfe1JAszJ43jtwEGWkP2686uGjTHRAf15nTVojCXlZwCP8KwJ5/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfh2YYkuyT5WJJ3D51FkiStjIVn214CXDF0CEmStHIWniUkOQB4CvDaobNIkqSVs/As7ZXAycD3xl2Z5MQkm5Js2rGxJEnSclh4JkjyVOC6qto8aZ2q2lhV66tq/Q6MJkmSlsnCM9ljgKcluRJ4M3BkktcPG0mSJK2EhWeCqvrdqjqgqg4EjgU+UFW/OHAsSZK0AhYeSZLUvHVDB5gHVXUBcMHAMSRJ0gp5hEeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeamqoTM0IYkPpCRpe9hcVeuHDjHv1g0dYN4l2QBsGDqHJEmazCM8ayRJVX1t6BgTJfsCMB8ZvzFwksmS+wBQ9e2Bk4yX7A5A1U0DJ5ks2QuAqg8OnGSy5HEAzPLf1NnUnd88bIwl3bM//6NBU0z2e/15HTFojCXlfMAjPGvCOTySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5q0bOsAsS3IlcBNwJ3BHVa0fNpEkSVoJC8+2HVFV1w8dQpIkrZxDWpIkqXkWnqUVcG6SzUlOHL0yyYlJNiXZNEA2SZI0JYe0lvbYqro6yf2B85J8qqouXLiyqjYCGwGS1FAhJUnS0jzCs4Squro/vw44Gzhs2ESSJGklLDwTJNkzyV4Ll4EnApcOm0qSJK2EQ1qT7QecnQS6x+mNVXXOsJEkSdJKWHgmqKovAD8+dA5JkrR6DmlJkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJal6qaugMTUjiAylJ2h42V9X6oUPMu3VDB5h3STYAG4bOIUmSJvMIzxpJUlXXDB1jomR/AMy4OlsyfmPgJOMl9wGg6ssDJ5kseSAAVS8fOMlkySkA1G8OHGQJeUV3Xh8aNsdS8tjufI9hY0x0a39eDx00xpJyBeARnjXhHB5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8S0iyd5KzknwqyRVJHj10JkmStHzrhg4w414FnFNVP5NkN2CPoQNJkqTls/BMkOTewOHACQBVdRtw25CZJEnSyjikNdmDgK8Bf5/kY0lem2TPoUNJkqTls/BMtg54BPCaqno4cAtwyuIVkpyYZFOSTUMElCRJ07HwTHYVcFVVXdT/+yy6AvRfqmpjVa2vqvU7PJ0kSZqahWeCqvoq8OUkB/eLjgIuHzCSJElaISctL+3FwBv6T2h9AXjOwHkkSdIKWHiWUFWXAA5XSZI05xzSkiRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWpeqmroDE1I4gMpSdoeNlfV+qFDzLt1QweYd0k2ABuGziFJkibzCM8aSVJV1wwdY6JkfwDMuDqznnFLvg8PnGSy5NEAVL1g4CSTJa8BoOqwgZNMlvwHAP934BxLeXF/PqvHJrKpO5/h3Uy/mz3CswacwyNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVngiQHJ7lk0enGJCcNnUuSJC3fuqEDzKqq+jRwKECSXYCrgbMHDSVJklbEIzzTOQr4fFV9ceggkiRp+TzCM51jgTeNLkxyInDijo8jSZKWwyM825BkN+BpwD+NXldVG6tqfVWt3/HJJEnStCw82/Yk4OKqunboIJIkaWUsPNv2bMYMZ0mSpPlh4VlCkj2Bo4G3DZ1FkiStnJOWl1BVtwD3GzqHJElaHY/wSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDUvVTV0hiYk8YGUJG0Pm6tq/dAh5t26oQPMuyQbgA3ArcAVa7z5fYDr13B79wa+tYbbg9nPuNb5YPYz7oz7GWY/4874XITZzzgPz8WD1nBbO6+q8rQGJ2DjdtjmJjPOVr55yLgz7ud5yLgzPhfnIePO+FzcWU/O4Vk77xo6wBTMuDZmPeOs5wMzrhUzrt6s54P5yDjzLDxrpKpm/glpxrUx6xlnPR+Yca2YcfVmPR/MR8Z5YOGZbRuHDjCFWc846/nAjGtl1jPOej4w41qZh4w7HT+lJUmSmucRHkmS1DwLjyRJap6FZwYlOT3JdUkuHTrLOEkemOT8JJcnuSzJS4bONCrJPZL8R5KP9xn/YOhM4yTZJcnHkrx76CzjJLkyySeTXJJk09B5xkmyd5KzknwqyRVJHj10psWSHNw/fgunG5OcNHSuUUl+o/9duTTJm5LcY+hMiyV5SZ/tsll5/Ma9Vie5b5Lzkny2P7/PkBm1hYVnNp0BHDN0iCXcAfxmVR0CPAp4YZJDBs406rvAkVX148ChwDFJHjVwpnFewtp/YeVaO6KqDq3Z/abXVwHnVNWPAD/OjD2eVfXp/vE7FHgk3ZeUnj1wrLtI8gDg14H1VfUwYBfg2GFTbZHkYcDzgMPo9vFTkzxk2FTA+NfqU4D3V9VBwPv7f2sGWHhmUFVdCNwwdI5Jquqaqrq4v3wT3RvMA4ZNdVfVubn/5679aaZm6Cc5AHgK8Nqhs8yrJPcGDgdOA6iq26rqm8OmWtJRwOer6otDBxljHbB7knXAHsBXBs6z2EOBi6rq1qq6A/hX4FkDZ5r0Wv104Mz+8pnAM3ZoKE1k4dGqJDkQeDhw0bBJttYPF10CXAecV1WzlvGVwMnA94YOsoQCzk2yOcmJQ4cZ40HA14C/74cGX5tkz6FDLeFY4E1DhxhVVVcDfwF8CbgG+FZVnTtsqru4FHhckvsl2QN4MvDAgTNNsl9VXdNf/iqw35BhtIWFRyuW5J7AW4GTqurGofOMqqo7+2GEA4DD+sPiMyHJU4Hrqmrz0Fm24bFV9QjgSXRDl4cPHWjEOuARwGuq6uHALczoEEKS3YCnAf80dJZR/TyTp9MVyO8H9kzyi8Om2qKqrgD+FDgXOAe4BLhz0FBTqO57X2bqyPLOzMKjFUmyK13ZeUNVvW3oPEvphzjOZ7bmRT0GeFqSK4E3A0cmef2wkbbW/+VPVV1HN+/ksGETbeUq4KpFR+/OoitAs+hJwMVVde3QQcZ4AvCfVfW1qrodeBvwkwNnuouqOq2qHllVhwPfAD4zdKYJrk2yP0B/ft3AedSz8GjZkoRuzsQVVfWXQ+cZJ8m+SfbuL+8OHA18athUW1TV71bVAVV1IN0wxweqamb+ogZIsmeSvRYuA0+kG1qYGVX1VeDLSQ7uFx0FXD5gpKU8mxkczup9CXhUkj363++jmLHJ30nu35//AN38nTcOm2iidwLH95ePB94xYBYtsm7oANpakjcBjwf2SXIV8LKqOm3YVHfxGOA44JP9HBmAl1bVewbMNGp/4Mwku9AV+7dU1Ux+9HuG7Qec3b3/sQ54Y1WdM2yksV4MvKEfMvoC8JyB82ylL4xHA88fOss4VXVRkrOAi+k+hfkxZu+/R3hrkvsBtwMvnIXJ6eNeq4GXA29J8lzgi8DPDpdQi/lfS0iSpOY5pCVJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl565a68phjjqnrr78eqJFrauR83LLl/Mw025h23dH1Zn3dSdcv52fWavvTbmvadaa8maWWreZntvf21+pndnSmoe7zcrY7RKah7/O2li1z89s7fqtPo+Vsf5rbnIWXoVnItKPu8/fgvVV1zJirli48119/PZs2bQLu6JdMOl/quoXzO1fwMytZd5qfWU2Wtc4/mmXIx2dSlmXcTt25ZdXvjWz2zgnLv8cW21p30eYnbmfSNrZXlnE/M+19nybLStZd6nFaavvTrruS/bCc7e+o/bCcLNPsh+30nJjFp1yrT6PlZNkZXlrWYj8vJ8tqX7pugX2YwCEtSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqXqpq8pXJOcA+Oy7OdrMPcP3QIbQi7rv55b6bb+6/+bUz77vrq+qYcVcsWXhakWRTVa0fOoeWz303v9x38839N7/cd+M5pCVJkppn4ZEkSc3bWQrPxqEDaMXcd/PLfTff3H/zy303xk4xh0eSJO3cdpYjPJIkaSdm4ZEkSc3bKQpPklOTXJ3kkv705KEzaduSHJPk00k+l+SUofNoekmuTPLJ/vdt09B5tLQkpye5Lsmli5bdN8l5ST7bn99nyIwab8K+8z1vjJ2i8PT+qqoO7U/vGTqMlpZkF+BvgCcBhwDPTnLIsKm0TEf0v29+H8jsOwMY/bK2U4D3V9VBwPv7f2v2nMHW+w58z9vKzlR4NF8OAz5XVV+oqtuANwNPHziT1KSquhC4YWTx04Ez+8tnAs/YoaE0lQn7TmPsTIXnRUk+0R/+89Ds7HsA8OVF/76qX6b5UMC5STYnOXHoMFqR/arqmv7yV4H9hgyjZcv+rtUAAAFOSURBVPM9b0QzhSfJ+5JcOub0dOA1wA8BhwLXAK8YNKzUvsdW1SPohiRfmOTwoQNp5ar7/hK/w2R++J43xrqhA6yVqnrCNOsl+X/Au7dzHK3e1cADF/37gH6Z5kBVXd2fX5fkbLohyguHTaVlujbJ/lV1TZL9geuGDqTpVNW1C5d9z9uimSM8S+l/WRc8E7h00rqaGR8FDkryoCS7AccC7xw4k6aQZM8key1cBp6Iv3Pz6J3A8f3l44F3DJhFy+B73njNHOHZhj9LcijdIdkrgecPG0fbUlV3JHkR8F5gF+D0qrps4Fiazn7A2Umge415Y1WdM2wkLSXJm4DHA/skuQp4GfBy4C1Jngt8EfjZ4RJqkgn77vG+523N/1pCkiQ1b6cY0pIkSTs3C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvP+P2TN+uaNyRa7AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "State Values for initial policy:\n",
            "\n",
            "[[-7.01 -7.04 -7.12 -7.17 -7.21 -7.22 -7.21 -7.16 -7.02 -6.71]\n",
            " [-6.96 -7.   -7.07 -7.15 -7.18 -7.2  -7.18 -7.13 -6.9  -5.73]\n",
            " [-6.88 -6.92 -7.   -7.12 -7.17 -7.18 -7.16 -7.11 -6.32 -2.26]\n",
            " [-6.79 -6.81 -6.9  -7.06 -7.11 -7.13 -7.12  0.    9.7   2.27]\n",
            " [-6.69 -6.7  -6.77 -6.95 -6.89 -5.73  6.75  9.07 13.18  6.23]\n",
            " [-6.61 -6.61 -6.63 -6.77 -6.48 -4.25  4.34  1.3  17.83  7.1 ]\n",
            " [-6.57 -6.56 -6.55 -6.54 -5.77 -3.43 -2.15 12.7   6.76  6.67]] \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h6fEfLvG7sys"
      },
      "source": [
        "# Task 2\n",
        "\n",
        "Starting from the initial policy, choose actions using ε-greedy action selection (ε=0.1)\n",
        "\n",
        "Improve the policy by SARSA(λ) for 10,000 episodes.\n",
        "\n",
        "Compute the resulting state value V(s) for each grid cell.\n",
        "\n",
        "Visualize the resulting policy using arrows."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFnmEyj6nZjN"
      },
      "source": [
        "## Epsilon Greedy Policy, Non-Deterministic Actions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aNWy0Vavot5V",
        "outputId": "1cd80e27-3303-43fe-b7b0-6596392f4e53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "Q = policyImprovement(deterministic=False)\n",
        "print()\n",
        "plotPolicy(Q,deterministic=False)\n",
        "print()\n",
        "V = np.max(Q, axis=2)\n",
        "plotStateValues(V)\n",
        "print()\n",
        "print('State Values for learned policy:\\n')\n",
        "print(np.round(V,2),'\\n')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 10000/10000 [05:14<00:00, 31.79it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAGqCAYAAADHtHSPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAWpklEQVR4nO3de4yld33f8c+ZPXvLMsauF8cOmBiHUpC4DMQGFK/wkEhtRSU7tKkSiVAaRaoqOc1meiFSL9pu1aitKmUzpfSfBkGEGqWtGoHa0hQpdKZNKG2y6VjEJhAb2xBjMOvY6/F6L97dp3/MjFmPZ9a7O+c8z/k+5/WyrNGes97z/WqOzrz9O5cdNE0TAIBJN9P1AAAAV0K0AAAliBYAoATRAgCUIFoAgBJECwBQwvByV87MzDQHDhxoa5bWnD59Okmyf//+jicZLXvV09fd7FVPX3ezV03PPffciaZpXrP58stGy4EDB7K6ujq+qToyPz+fJFlaWup0jlGzVz193c1e9fR1N3vVNBgMHtvqck8PAQAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUMGiaZtsrh8Nhc+jQoRbHacfKykqSZG5uruNJRste9fR1N3vV09fd7FXT8vLy8aZp7th8uZMWAKCEy560zM7ONqurqy2O0475+fkkydLSUqdzjJq96unrbvaqp6+72aumwWDgpAUAqEu0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUMGiaZtsrh8Nhc+jQoRbHacfKykqSZG5uruNJRste9fR1N3vV09fd7FXT8vLy8aZp7th8uZMWAKCEy560zM7ONqurqy2O0475+fkkydLSUqdzjJq96unrbvaqp6+72aumwWDgpAUAqEu0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoIRB0zTbXjkcDptDhw61OE47VlZWkiRzc3MdTzJa9qqnr7vZq56+7mavmpaXl483TXPH5suHl/uPLlxMvvQn4xuqK+fOJoNB11MATJZnczbLebTrMUbsTAbxgN8Xlz1pmdk329z6L1ZbHKcd3/nYfJLkzENLnc4xavPz80mSpaWlTucYtb7ulfR3N3vVMz8/vxYsS3+961FGa/5TSZJm6dFOxxi1Pt8Xk2QwGGx50uI1LQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAlDDu99abJLY+dzhv/8Nnse/5CVq/fnQd/+Po8e+OeTscCACZPZ9Hymm+dyT2f+kZmn3khu881GSS5sCv5kc9/N9984/flP3/41pw50G1TAQCTo5Onhw5+60w+tPj13PjkuexZD5Yk2XUhGZ5vcusfn8qHj309e05f6GI8AGACdRIt9/zaN7Pn7MVtrx9eSGaffiGH/tt3WpwKAJhkrUfLzd84neuePveKv294ocnbv/RMhue2jxsACmq6HoCqWo+W2x94NsNzV3aPbWaSH3js+TFPBEBbZs4P8pGffnd+7F++qetRKKj1V7ruO33xikupSbL7Mk8jAVDHzPlBPvyRO3Pnr78+d/7665Mkv/33vtbxVFTS+knL6vXDnN81eOXfmGTmYnLqOu8gAqju0mBp1v/54Eff7sSFq9J6tHzlXdcnV9YsObt/Jt++df94BwJgrDYHy2D9H+HC1Wo9Wp67fne+/pZX5fzw8uXSJDl+6M8kgyssHAAmzlbBskG4cLU6ecvz5z70ujxz455tw+Vi1g5j3vZ7z+TAyRdanQ2A0bhcsGwQLlyNTqLl3L5d+fTfvj3H33djzu6bydm9Mzmzbybn9szk5A27s3TP9+fJW/bmxifP5ac+/qhwASjmSoJlg3DhSnX2KtcX9u7K8j0353c+cFNu/sbp7Dl7Madmh3nytfuSwSAPvPuG/OTHH8lNT5zNT3380fzGfbfl1Kt3dzUuAFfoaoJlw6XhknhXEVvr/G95vjCcyeO3H8gjb5nNk6/b/+JrWE6/aph/f98b1k9czjpxASjiQz/7wy++pflKgmXDxu/94Effnrv/1RvHMhu1dR4tlyNcAOrZd3Lnp+J7T+0awST0zcR/CMpGuHiqCKCGz/3jB/PEW5/d8rqDDx/IHb+xdgrzW//gK9v+GV/82UfHMRrFTXy0JMIFoJLH507m8bmTW173+t+/4cVo+S//9IE2x6IHJvrpoUt5qggApluZaEmECwBMs1LRkggXAJhW5aIlES4AMI1KRksiXABg2pSNlkS4AMA0KR0tiXABgGlRPloS4QIA06AX0ZIIFwDou95ESyJcAKDPehUtiXABgL7qXbQkwgUA+qiX0ZIIFwDom95GSyJcAKBPeh0tydbhsutC0/VYAMBV6n20JC8Pl5sePyNcAKCYQdNs/8N7MDNs9t5+qMVxxmvmQpObvnUm3zn3YJrBIHe+953ZtWfQ9Vgjs7KykiSZm5vreJLR6uteSX93s1c9KysrOZkzydzNY72dvavD3PoHNyRJHrr7u2O9rSTJyreTJHfPvXf8t9WiPt8Xk2R5efl40zR3bL58Kk5aNlzcNciTP7AvzWCQQdPkqZUzuXDOiQtAkgzSn/+Ju1Rf95pGw8td+aoD+7P60FJLo7TnfXfdnRP3n8l9pz6RG57em3u/cFsO3LK767F2bH5+PkmytLTU6Ryj1te9kv7uZq962trtyd8/nf9458NJkvuW3jrW20r6+z3r614bBoOtQ3OqTlo2zOwe5OA79uXGt+3N0390Np/90Udz6gnvKgKASTaV0ZKshcu9X3iDcAGAIqY2WpJk/8GhcAGAIqY6WhLhAgBVTH20JMIFACoQLeuECwBMNtFyCeECAJNLtGwiXABgMomWLQgXAJg8omUbwgUAJotouQzhAgCTQ7S8AuECAJNBtFwB4QIA3RMtV0i4AEC3RMtVEC4A0B3RcpWECwB0Q7RcA+ECAO0TLddIuABAu0TLDggXAGiPaNkh4QIA7RAtIyBcAGD8RMuICBcAGC/RMkLCBQDGR7SMmHABgPEQLWMgXAC29q3/+VzXI1CYaBkT4QLwUg//5sl88aPfSZLc9O79HU9DRaJljIQLwJqHf/NkPv+T30xzIXnXLx7MT3zp9q5HoiDRMmbCBZh2G8Fy8fxasLz3n31/BoNB12NRkGhpgXABppVgYZRES0uECzBtBAujJlpaJFyAaSFYGAfR0jLhAvSdYGFcREsHhAvQV4KFcRItHREuQN8IFsZNtHRIuAB9IVhog2jpmHABqhMstEW0TADhAlQlWGiTaJkQwgWoRrDQNtEyQYQLUIVgoQuiZcIIF2DSCRa6IlomkHABJpVgoUuiZUIJF2DSCBa6JlommHABJoVgYRKIlgknXICuCRYmhWgpQLgAXREsTBLRUoRwAdomWJg0oqUQ4QK0RbAwiURLMcIFGDfBwqQSLQUJF2BcBAuTTLQUtVW4XDjXdD0WUNjpE+cFCxNNtBS2OVyeWjkjXIBrcvrE+Tz9wFnBwkQbNM32P+SGw2Fz6NChFsdpx8rKSpJkbm6u40lG4+ILTU7cfyaPnHowMzODvOs978yuPf15sOnb9+tSfd3NXrWcPnE+X37g/jRJ3nLr23Ld7Xu6Hmlk+vo96+teG5aXl483TXPH5sudtPTAzO5BDr5jX2ZmBrl4sXHiAlyxjROWJslw70yvgoX+uexJy+zsbLO6utriOO2Yn59PkiwtLXU6x6i97667c+L+M7nv1Cdyw5v35t4v3JYDt+zueqwd6+v3K+nvbvaq4dIX3X7i1r+R627f05vdNvTte7ahr3ttGAwGTlr6buPExbuKgFey+V1CTlioQLT0zMzugbdDA5e11duaoQLR0kM+xwXYjs9hoTLR0lPCBdhMsFCdaOkx4QJsECz0gWjpOeECCBb6QrRMAeEC00uw0CeiZUoIF5g+goW+ES1TRLjA9BAs9JFomTLCBfpPsNBXomUKCRfoL8FCn4mWKSVcoH8EC30nWqaYcIH+ECxMA9Ey5YQL1CdYmBaiBeEChQkWpoloIYlwgYoEC9NGtPAi4QJ1CBamkWjhJYQLTD7BwrQSLbyMcIHJJViYZqKFLQkXmDyChWknWtiWcIHJIVhAtPAKhAt0T7DAGtHCKxIu0B3BAt8jWrgiwgXaJ1jgpUQLV2xzuHzm/Y8IFxgTwQIvJ1q4Ki+Gy9v35ZmvnnPiAmMgWGBrooWrtv/gMPf+9m2eKoIxECywPdHCNfEaFxg9wQKXJ1q4ZsIFRkewwCsTLeyIcIGdEyxwZUQLOyZc4NoJFrhyooWREC5w9QQLXB3RwsgIF7hyggWunmhhpIQLvDLBAtdGtDBywgW2J1jg2okWxkK4wMsJFtgZ0cLYCBf4HsECOydaGCvhAoIFRkW0MHbChWkmWGB0RAutEC5MI8ECoyVaaI1wYZoIFhg90UKrhAvTQLDAeIgWWidc6DPBAuMjWuiEcKGPBAuMl2ihM8KFPhEsMH6ihU4JF/pAsEA7RAudEy5UJligPaKFiSBcqEiwQLtECxNDuFCJYIH2iRYmylbhcuFc0/VY8BKCBbohWpg4m8PlqZUzwoWJcfrEecECHRk0zfY/DIbDYXPo0KEWx2nHyspKkmRubq7jSUarb3tdfKHJifvP5JFTD2ZmZpB3veed2bWnXz8c+vY929DXvf7g9/5fzj9/Ma/NmzN76+5cd/uerkcamb5+z+xV0/Ly8vGmae7YfLmTFibWzO5BDr5jX2ZmBrl4sXHiQqdOnzif889fTJP0LligisuetMzOzjarq6stjtOO+fn5JMnS0lKnc4xaX/d6311358T9Z3LfqU/khjfvzb1fuC0Hbtnd9Vgj0dfvWd/22ngNyy+f/5nM3ro7xx/73d49JdS379kGe9U0GAyctFDTxomLdxXRhUtfdLtxwtK3YIEqRAslzOweeDs0rdv8LiFPCUG3RAtl+BwX2rTV25qBbokWShEutMHnsMBkEi2UI1wYJ8ECk0u0UJJwYRwEC0w20UJZwoVREiww+UQLpQkXRkGwQA2ihfKECzshWKAO0UIvCBeuhWCBWkQLvSFcuBqCBeoRLfSKcOFKCBaoSbTQO8KFyxEsUJdooZeEC1sRLFCbaKG3hAuXEixQn2ih14QLiWCBvhAt9J5wmW6CBfpDtDAVhMt0EizQL6KFqSFcpotggf4RLUwV4TIdBAv0k2hh6giXfhMs0F+ihakkXPpJsEC/iRamlnDpF8EC/SdamGrCpR8EC0yHYdcDQNc2wuWzP/pInvry2Xxm/pH84AdmW7ntkw+dS5L8zsITrdxeW9rc6/yZJl/51T8VLDAFRAvk5eHyzNeeauV2n8vaqc79v9LO7bWli70EC/SfaIF1G+Hy1U8/09ptfvLje5Ikd913c2u32Ya299r/mmHe9KFXCxboOdECl9h/cJi5hYOt3d6rPrs7SVq9zTb0dS+gW16ICwCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUMGiaZvsrB4PvJnmsvXEAAPKDTdO8ZvOFl40WAIBJ4ekhAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEoZdD7CdhaOLr0vyT5L8xSQ3JnkiyWeSHD125PDTXc62E/aqZeHo4k8kuTvJXJJ3JJlN8u+OHTn8050OtkP2qmXh6OKNST6Y5C8leVuS1yY5l+TLST6Z5JPHjhy+2N2E16aveyX9vS9u6OoxfyJPWhaOLv5QkuNJfibJ/01yLMnXkxxO8r/X7+jl2Kukf5jk57L2wPN4x7OMkr1q+atJ/m2S9yT5P0l+Jcl/SvLWJL+a5D8sHF0cdDfeNevrXkl/74udPuZP6knLv0lyU5KfP3bk8Mc2Llw4uvjLSRaS/FKSv9nRbDthr3oWkvxJkoey9n9N/6PbcUbGXrV8Lck9Sf7rpScPC0cX/37Wfmj8lSR/OWs/8Cvp615Jf++LSYeP+YOmacbx516z9YJ7KMmjSX5o0x15NmtHUIMkNx07cvhUJ0NeA3vV2msrC0cX57P2wNObI97EXtWt/4D/pST/+tiRw3+r63lGpU979em+2PVj/iQ+PfT+9a+f3/xc5rEjh1eT/G6S70vy3rYH2yF7AePwwvrX851OMXp93au6Th/zJzFa/tz6169tc/0fr399UwuzjJK9gJFaOLo4TPLX1n/5W13OMkp93asnOn3Mn8RoefX615PbXL9x+fUtzDJK9gJG7Z9n7UWrnzt25PB/73qYEerrXn3Q6WP+JEYLAK9g4ejizyf5O0n+KMmHOx5nZPq6F6MxidGyUWmv3ub6jcufaWGWUbIXMBILRxd/LslikgeTvP/YkcN/2vFII9HXvXqm08f8SYyWr65/3e75sD+7/nW759Mmlb2AHVs4uvgLST6W5A+z9oP92x2PNBJ93auHOn3Mn8Ro2Xgv+59fOLr4kvnW3051V5Lnk3yp7cF2yF7AjiwcXfzFrH2Q10rWfrA/2fFII9HXvXqq08f8iYuWY0cOP5zk80luS3LfpquPJjmQ5NPVPvPDXrX2gkmzcHTxH2XtBarHk/zYsSOHT3Q80kj0da++6voxf+I+XC558cNrvpi1T9z7bJKvZO1jnt+ftSOnHzl25PBT3U14bexVz8LRxR9P8uPrv7w5yV/I2sdV/6/1y04cO3L473Yx207Yq5aFo4sfSfKpJBey9hTKVu/cePTYkcOfanGsHevrXkl/74tJt4/5E/kx/seOHH544ejiHfneX8b0gax9yt5iCv8FfPYqaS7JRzZddvv6v0nyWJKKDzz2quUN6193JfmFbX7PctYCoJK+7pX0977Y6WP+RJ60AABsNnGvaQEA2IpoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEASvj/bLka1UTf2AgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAGSCAYAAADn8RuZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3debQkdX338fdHBhRQQQUJggaNBCU+EXRCJBoiIAaXcXuyYJSoMWKMUfFJQtQkJ/h4spjF7ZiYMw+oGAkuCLgkQVAxaILEGcCFxTWoKLKILILK9n3+qJpwabrv9L1zZ6r7N+/XOX2qb1V19ae7uu/9TNWve1JVSJIktexuQweQJEna3Cw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRGpXkk0kuHTrHNJLskOQtSb6Z5LZ5yb2pkhybpJLsNXSWSZK8M0mNzJv53NIoC4/mSpKHJFmb5JIkNyX5fpKLk5yQ5OCRdY9N8owVuM+jkzx/U7czZrtP6f9ovGkj6722X+83VjrDDPkj4GXAe4HnA0cPmmYO9aXx6CSfSnJNkluSXJHkX5M8P8mqoTNKQ/INoLmRZDXw78AtwLuAC4Htgb2BJwI3AGctuMmfAScAp23iXR8NXAq8cxO3M+p04DvAc5L8YVXdMrpCkgDPA64FTlnh+58lhwFfqKo/HDrIPEryUOBfgJ8GPgb8JXA1cH/gCcA7gH2BY4bKKA3NwqN58mfADsB+VfW50YVJfmLLR1q+qrotyTuB1wBrGF9oDgF+EviHqvrRFoy3pf0E8M2V3miSe1XVDSu93VmSZHvgI8BDgP9dVaOvo9cn+Tng57Z4OGmGeEpL82Rv4Hvjyg5AVX0XIMleC8YcPK8/HVQLxyEk+fUkH+rHjPw4ydVJTkvyswu32d/mJ4FfWridhWMXkqxOcmq/jR8n+VKSP57yFMLbgQJ+a8Ly31qw3tS5J0lyaZJPjpn/+P5xPX9k/t2TvCbJhUl+lOTaJB9Osv/IenfrT6d8PskNSa7vn4fjk2y7SJ7n98/xg7nzc3zsgnWekeQ/ktyY5Af99adPemxJ9k/y0STXAZ+f4jnZPcnb+uf05iTf6U+b3n9kvQck+bskF/SnUn+U5KIkf5RkmzHb3S7JMf36NyW5Lsm6JL83Jsbdk/xFksv6/fq5JE/eWPbebwP7AH83puwAUFWfrap/GMm3Ka9bae74wtY8+RqwT5JnTfrF3rsKOBL4J+BTwNox6/we8L1+2XeBnwKOAv4jyaOq6iv9ekcCb6Q7PfDnI/dBkqfQHZn5KvB3wDXAgcD/BfYDfnWxB1RVX0tyNnB4kt2r6vINy5LsBDwT+FxVrV9i7k3WF5XTgV+gey7fCuwEvKi/v4Oqal2/+h/TPeYPA/8I3EZXYp4G3J3uNOQ4ZzP+Of58n+F3gb8HLum3D90Yn9OSvLiqRvftg4BPAO8HPgDccyOP8UHAOcB2wPF0r7GHAi8BDk6yuqqu61f/WeBZwKn9etsChwN/RXd05cULtrsd8FHg8cAZwLuBHwH/q9/GW0einED3HP1tn+Xo/jH+dFVduthjAH6ln457nU963Jv0upXmUlV58TIXF7pfyDfTHRH5Mt1Rj5cAD5+wfgHvnLBsxzHzHg78mO700cL5lwKfHLP+PehKx9nAqpFlr+zv//FTPK4j+3WPGZn/4n7+y5eZ+5PApVM+lsf39/X8MY/hl0fWvTfd6adPLph3HnDRJuzbu+QC7gP8gO6P8r1H7v9rdGO2dh7ZRgG/vYT7/SBwJbDnyPzVwK3AsQvmbQ9kzDb+ia7g7b5g3jF9lr8Ys/7dFlw/tl/vIwu3TXf6qYC/nOIxfA+4bgmPeUmvW7qxazWy3obcey13n3vxsqUvntLS3Kiqc4BH0/1reCfgBcA/ABclOTvJQ5awrRuhGxSc5N5JdqE7avMl4Oen3MxhwG50A0J3TrLLhgvwr/06T5xiOycD1/WPZ6EX0BWZd69w7mk9l+7IyvqRx7YdcCbwuHTjR+jz75HkcSt4/4cBOwJvqarrN8zsr7+F7ujNE0Zucw3d/tio/gjaU4EPAT8aeYyX0hWt/9l/VfXDqqr+ttsluW+/7kfphgesXrD55wDf546jUizYzu1j4rx5w7b7dT5LV/b2nuKh3Juu/E1rpV630lzxlJbmSlV9ge6UBkl+EvglujEMvwh8MMmjq+rmjW2nH4PyOrojGzuOLP7vKeM8vJ++fZF1dtvYRqrqh0lOAn4nyYFVdU6SfekKzPuq6poVzj2th9Md1bhqkXV2Ab5FN/D6NOBTSb5Dd3TpX4CTp9kfEzy4n144ZtmGeaMl92tVdduU29+Hrqi8sL+M8/UNV/qxLa8CfpPutFdG1r3Pgut7AxfU9APNvz5m3veA+01x2+uBe015P7BCr1tp3lh4NLeq6hvAu5JsGKvzWOAA4NOL3a4ft3E23R+K19EdHbmR7hD9m9jIuI+Fm+qnfwhcMGGd70y5rbcDv0N3VOcc7jjac/wK564J88f9LgjwBeD/LLK9q6A7+pbkp4BfBg7uL78B/EmSxy0sbZvZTUtYd8P+ezfdUcNxfrjg+hu447uC/pzuVNgtwKOA17NpHwKZVNJGS9U4XwQOSvKQqhpXnCZtcyVet9LcsPBo7lVVJTmXrvDsMcVNnklXDp5WVQu/t4ck96M7jXSnu5iwnQ0DhG+sqo8tIfJdVNVnk3wB+PUkf0A3ruebdN+pstzc41wD3HfM/HGnA78C7Ap8YsJpmNHH8AO6gcIf6DNtGHD8QuBvpsg2asMf758BPj6ybN+RdZbjq3T7drsp99+RwNlVdcTCmem+A2fUl4GHJbl7VU2zXzbFB4CD6I50vmaK9VfsdSvNE8fwaG4kOWzcR2b7cSQbxhxctGDRDxj/x33Dv6bv9K/nJC+i+z6YUZO281G6f+W/KsldlifZPslSTjUcTzce4zi6UwrvHCkaS809zoY/xP9TDJPcHXjpmHXf1W937BGeJLstuL7LmFXO66fjnrtpnEl3BOtlC5/H/vrL6PbLmcvcNlX1PboxK89K8pjR5f04qV0XzLqNuz73O9IN9B11It0prj8Zt93lZp7gOLqjfX+QMR/X7+/z0X0BhZV/3UpzwSM8midvBO6X5EN0p1puAh5Id+rkp4F39WN8NvgM8IQkf0R3tKSq6j3Av/W3/ackb6UbXPpY4Ml0n/4ZfV98BnhhktcBFwO3Ax+uqhuT/Cbd2JUvJXk73VGDnYGH0X38+Jl041mm8W7gr+k+ElzcdfDtUnOP81bgCOBjSf6RbgDykYw/FfRmugGuf5PkELqPe19P99HvQ+k+Zr3hv/O4OMlngHPpTofsTvdx+ZuB90yR6y6q6tokx9AdJTo33Zc0QjeG66HAi+uOj4wv10voToGeneRdwPl0/xB8CPB0utJ3bL/uycCLk7yX7sjbbnTfk/S9Mdt9M92XSf5Jui/9O4Pu+foZurFDo4Otl62qbkryVLoxU6clOYOuCH6P7gjdwXSnGv+6X3+lX7fSfBj6Y2JevEx7oTuK8/fA5+i+s+VWul/qZ9H94bnbyPp70/2huZ6uQNSCZQfR/aG7ge6/bfgX4BGM/yj3/elOG1xDV3bu9HHc/nbvBr5N9wf+CuA/gT8F7rvEx/jefvsfn7B8KbnvMq+f/zy6IwI30w10PobuG53v9LH0ft1VwMuBz9IdbbmR7pTIicATF6z3KrrxRVfSnVr7Ft134Txqysd9KWM+Lt8ve2b/fG64//8EnrGUbWzkvnehO+X2ZbpSci1doX4zsO+C9Xbo1/tGv95X+sd96ITn7h5030904YLtfhb43QXrHDv6elru4+nzvbJ/fXyfbnzRFf1r5Ehgm5H1p3rd4sfSvTRySdWk4QmSJEltcAyPJElqnoVHkiQ1z8IjSZKaZ+GRJEnN82PpmyjJGrqPn75o6CySpCZdXVW7bnw1LcZPaa2QJDXLz+WG7zqbj4y3DJxksmRbYHafxzuew6X8X5Jb1obvtKv61MBJJkt+EYCqPx04yWTd10JB1cEbWXM4SfeF4LP6lu7fztRXh82xmP57vNdX1eqNrKqN8JSWJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLzwRJ3p7kyiRfHDqLJEnaNBaeyd4JHD50CEmStOksPBNU1dnANUPnkCRJm27V0AHmWZKjgKOGziFJkhZn4dkEVbUWWAuQpAaOI0mSJvCUliRJap6FR5IkNc/CM0GSk4BzgH2SXJbkhUNnkiRJy+MYngmq6tlDZ5AkSSvDIzySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmpeqGjpDE5L4REqSNof1VbV66BDzbtXQAeZdkjXAmqFzSJKkyTzCs0KS1Cw/l0kAqLpl4CSTJdsCZtwUs54PFmY8beAkkyXPAKDqlQMnmSx5IwBVvzBwksmS/wSgvjNwkAnygG5aHxs2x2LyBMAjPCvCMTySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JkjywCRnJbkoyYVJXjF0JkmStDyrhg4ww24Ffr+qzktyL2B9kjOr6qKhg0mSpKXxCM8EVXV5VZ3XX78BuBjYY9hUkiRpOSw8U0iyF7A/cO6wSSRJ0nJ4SmsjktwT+ABwdFVdP7LsKOCoQYJJkqSpWXgWkWRburJzYlWdMrq8qtYCa/t1awvHkyRJU/KU1gRJAhwPXFxVbxg6jyRJWj4Lz2SPBY4EDklyQX958tChJEnS0nlKa4Kq+jSQoXNIkqRN5xEeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOalqobO0IQkPpGSpM1hfVWtHjrEvFs1dIB5l2QNsGboHJIkaTKP8KyQJFX1/aFjTJTcB4BZ3t9JgHnJeMvAScZLtgWg6qqBk0yW7ApA1WkDJ5kseQYAVa8eOMlkyV8CUPWAgZNMlnwHgDp34CAT5Oe76aeHjbGox3UTj/CsAMfwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZkgyT2S/FeSzyW5MMlrh84kSZKWZ9XQAWbYj4FDquoHSbYFPp3k36rqM0MHkyRJS2PhmaCqCvhB/+O2/aWGSyRJkpbLU1qLSLJNkguAK4Ezq+rckeVHJVmXZN0wCSVJ0jQsPIuoqtuqaj9gT+CAJI8YWb62qlZX1ephEkqSpGlYeKZQVdcCZwGHD51FkiQtnYVngiS7Jtm5v749cBhwybCpJEnScjhoebLdgROSbENXDN9XVR8ZOJMkSVoGC88EVfV5YP+hc0iSpE3nKS1JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5qWqhs7QhCQ+kZKkzWF9Va0eOsS8WzV0gHmXZA2wZugckiRpMo/wrJAkVfWioWNMlPw/AKreOnCSyZLfA6DqmQMnmSw5FYCqHw6cZLxkewCqvjJwksmSvQGoes/ASSZLjgCg6gUDJ5kseQcAVecMnGSy5EAAdhg4xyQ39dP600FjLCqvAzzCsyIcwyNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVnI5Jsk+T8JB8ZOoskSVoeC8/GvQK4eOgQkiRp+Sw8i0iyJ/AU4Lihs0iSpOWz8CzuTcAxwO3jFiY5Ksm6JOu2bCxJkrQUFp4JkjwVuLKq1k9ap6rWVtXqqlq9BaNJkqQlsvBM9ljgaUkuBd4DHJLk3cNGkiRJy2HhmaCqXl1Ve1bVXsARwCeq6rkDx5IkSctg4ZEkSc1bNXSAeVBVnwQ+OXAMSZK0TB7hkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWpeqmroDE1I4hMpSdoc1lfV6qFDzLtVQweYd0nWAGuGziFJkibzCM8KSVJVbxo6xkTJ0QBUze5ZzOR2AOqagYMsIvftpnXOsDkmyYHdtH5z2ByLybu66WXDxljUnv10lv9Jva6fPnTQFIv7aj994qApJjujn9b5g8ZYVPYHPMKzImb3r58kSdIKsfBIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNW/V0AFmWZJLgRuA24Bbq2r1sIkkSdJyWHg27uCqunroEJIkafk8pSVJkppn4VlcAWckWZ/kqNGFSY5Ksi7JugGySZKkKXlKa3GPq6pvJ7k/cGaSS6rq7A0Lq2otsBYgSQ0VUpIkLc4jPIuoqm/30yuBU4EDhk0kSZKWw8IzQZIdk9xrw3XgicAXh00lSZKWw1Nak+0GnJoEuufpn6vq9GEjSZKk5bDwTFBVXwceOXQOSZK06TylJUmSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5qaqhMzQhiU+kJGlzWF9Vq4cOMe9WDR1g3iVZA6wZOockSZrMIzwrJElVXTV0jImSXQGoOmDgJJMl/wVAfWngIIvIPt20zho2xyQ5uJvWW4bNsZi8vJvW0cPmWEze1E0fOWyMRX2un1btO2iOxSQXAXDgwDkmOaef1tpBYywqRwEe4VkRjuGRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/Cs4gkOyc5OcklSS5OcuDQmSRJ0tKtGjrAjHszcHpV/UqS7YAdhg4kSZKWzsIzQZKdgIOA5wNU1c3AzUNmkiRJy+MprckeDFwFvCPJ+UmOS7Lj0KEkSdLSWXgmWwU8CnhbVe0P3Ai8auEKSY5Ksi7JuiECSpKk6Vh4JrsMuKyqzu1/PpmuAP2PqlpbVauravUWTydJkqZm4Zmgqr4LfCvJPv2sQ4GLBowkSZKWyUHLi3sZcGL/Ca2vAy8YOI8kSVoGC88iquoCwNNVkiTNOU9pSZKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDUvVTV0hiYk8YmUJG0O66tq9dAh5t2qoQPMuyRrgDVD55AkSZN5hGeFJKmqc4aOMVFyIABVvzpwksmS9wNQVwwcZBHZrZvWvw2bY5I8qZvWicPmWEye002vHjbGonbppy8ZNMXi3tZP5+L3zr8PHGSC/FI3rXr2sEEWkZwEHuFZEY7hkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwjNBkn2SXLDgcn2So4fOJUmSlm7V0AFmVVV9CdgPIMk2wLeBUwcNJUmSlsUjPNM5FPhaVX1j6CCSJGnpPMIznSOAk0ZnJjkKOGrLx5EkSUvhEZ6NSLId8DTg/aPLqmptVa2uqtVbPpkkSZqWhWfjngScV1VXDB1EkiQtj4Vn457NmNNZkiRpflh4FpFkR+Aw4JShs0iSpOVz0PIiqupG4H5D55AkSZvGIzySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzUtVDZ2hCUl8IiVJm8P6qlo9dIh5t2roAPMuyRpgDXATcPEKb34X4OoV3N5OwHUruD2Y/YwrnQ9mP+PWuJ9h9jNuja9FmP2M8/Ba3HsFt7X1qiovK3AB1m6Gba4z42zlm4eMW+N+noeMW+NrcR4ybo2vxa314hielfPhoQNMwYwrY9Yzzno+MONKMeOmm/V8MB8ZZ56FZ4VU1cy/IM24MmY946znAzOuFDNuulnPB/ORcR5YeGbb2qEDTGHWM856PjDjSpn1jLOeD8y4UuYh41bHT2lJkqTmeYRHkiQ1z8IjSZKaZ+GZQUnenuTKJF8cOss4SR6Y5KwkFyW5MMkrhs40Ksk9kvxXks/1GV87dKZxkmyT5PwkHxk6yzhJLk3yhSQXJFk3dJ5xkuyc5OQklyS5OMmBQ2daKMk+/fO34XJ9kqOHzjUqySv798oXk5yU5B5DZ1ooySv6bBfOyvM37nd1kvsmOTPJV/rpfYbMqDtYeGbTO4HDhw6xiFuB36+qfYHHAC9Nsu/AmUb9GDikqh4J7AccnuQxA2ca5xWs/BdWrrSDq2q/mt1ven0zcHpVPQx4JDP2fFbVl/rnbz/g0XRfUnrqwLHuJMkewMuB1VX1CGAb4IhhU90hySOAFwEH0O3jpyZ56LCpgPG/q18FfLyq9gY+3v+sGWDhmUFVdTZwzdA5Jqmqy6vqvP76DXR/YPYYNtWdVecH/Y/b9peZGqGfZE/gKcBxQ2eZV0l2Ag4Cjgeoqpur6tphUy3qUOBrVfWNoYOMsQrYPskqYAfgOwPnWejhwLlVdVNV3Qr8O/CsgTNN+l39dOCE/voJwDO2aChNZOHRJkmyF7A/cO6wSe6qP110AXAlcGZVzVrGNwHHALcPHWQRBZyRZH2So4YOM8aDgauAd/SnBo9LsuPQoRZxBHDS0CFGVdW3gb8FvglcDlxXVWcMm+pOvgj8YpL7JdkBeDLwwIEzTbJbVV3eX/8usNuQYXQHC4+WLck9gQ8AR1fV9UPnGVVVt/WnEfYEDugPi8+EJE8Frqyq9UNn2YjHVdWjgCfRnbo8aOhAI1YBjwLeVlX7Azcyo6cQkmwHPA14/9BZRvXjTJ5OVyAfAOyY5LnDprpDVV0MvB44AzgduAC4bdBQU6jue19m6sjy1szCo2VJsi1d2Tmxqk4ZOs9i+lMcZzFb46IeCzwtyaXAe4BDkrx72Eh31f/Ln6q6km7cyQHDJrqLy4DLFhy9O5muAM2iJwHnVdUVQwcZ4wnAf1fVVVV1C3AK8AsDZ7qTqjq+qh5dVQcB3we+PHSmCa5IsjtAP71y4DzqWXi0ZElCN2bi4qp6w9B5xkmya5Kd++vbA4cBlwyb6g5V9eqq2rOq9qI7zfGJqpqZf1EDJNkxyb02XAeeSHdqYWZU1XeBbyXZp591KHDRgJEW82xm8HRW75vAY5Ls0L+/D2XGBn8nuX8/fRDd+J1/HjbRRB8Cntdffx7wwQGzaIFVQwfQXSU5CXg8sEuSy4A/q6rjh011J48FjgS+0I+RAXhNVf3rgJlG7Q6ckGQbumL/vqqayY9+z7DdgFO7v3+sAv65qk4fNtJYLwNO7E8ZfR14wcB57qIvjIcBLx46yzhVdW6Sk4Hz6D6FeT6z998jfCDJ/YBbgJfOwuD0cb+rgb8C3pfkhcA3gF8bLqEW8r+WkCRJzfOUliRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOatWmzh4YcfXldffTVQI0tqZDpu3lJuM802pl13dL1ZX3fS8qXcZqW2P+22pl1nyrtZbN6m3GZzb3+lbrOlMw31mJey3SEyDf2YNzZviZvf3PFbfRktZfvT3Ocs/BqahUxb6jHfDh+tqsPHLFq88Fx99dWsW7cOuLWfM2m62LIN09uWcZvlrDvNbTYly0rnH80y5PMzKcsS7qduu2PV20c2e9uE+bdzh42tu2DzE7czaRubK8u420z72KfJspx1F3ueFtv+tOsuZz8sZftbaj8sJcs0+2EzvSZm8SXX6stoKVm2hl8tK7Gfl5JlU3913Qi7MIGntCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXmpqskLk9OBXbZcnDvZBbh6oPvW5uW+bZf7tl3u23a1tG+vrqrDxy1YtPAMKcm6qlo9dA6tPPdtu9y37XLftmtr2bee0pIkSc2z8EiSpObNcuFZO3QAbTbu23a5b9vlvm3XVrFvZ3YMjyRJ0kqZ5SM8kiRJK8LCI0mSmjeThSfJ4Um+lOSrSV41dB4tX5IHJjkryUVJLkzyin7+fZOcmeQr/fQ+Q2fV0iXZJsn5ST7S//zgJOf27933Jtlu6IxaniQ7Jzk5ySVJLk5yoO/bNiR5Zf/7+ItJTkpyj63hvTtzhSfJNsDfA08C9gWenWTfYVNpE9wK/H5V7Qs8Bnhpvz9fBXy8qvYGPt7/rPnzCuDiBT+/HqPCO6MAAAKUSURBVHhjVT0U+D7wwkFSaSW8GTi9qh4GPJJuP/u+nXNJ9gBeDqyuqkcA2wBHsBW8d2eu8AAHAF+tqq9X1c3Ae4CnD5xJy1RVl1fVef31G+h+ae5Bt09P6Fc7AXjGMAm1XEn2BJ4CHNf/HOAQ4OR+FffrnEqyE3AQcDxAVd1cVdfi+7YVq4Dtk6wCdgAuZyt4785i4dkD+NaCny/r52nOJdkL2B84F9itqi7vF30X2G2gWFq+NwHHALf3P98PuLaqbu1/9r07vx4MXAW8oz9leVySHfF9O/eq6tvA3wLfpCs61wHr2Qreu7NYeNSgJPcEPgAcXVXXL1xW3Xcj+P0IcyTJU4Erq2r90Fm0WawCHgW8rar2B25k5PSV79v51I+7ejpdqX0AsCMw9v+eas0sFp5vAw9c8POe/TzNqSTb0pWdE6vqlH72FUl275fvDlw5VD4ty2OBpyW5lO608yF0Yz527g+Tg+/deXYZcFlVndv/fDJdAfJ9O/+eAPx3VV1VVbcAp9C9n5t/785i4fkssHc/Ynw7usFUHxo4k5apH9dxPHBxVb1hwaIPAc/rrz8P+OCWzqblq6pXV9WeVbUX3Xv0E1X1HOAs4Ff61dyvc6qqvgt8K8k+/axDgYvwfduCbwKPSbJD//t5w75t/r07k9+0nOTJdOMDtgHeXlV/PnAkLVOSxwGfAr7AHWM9XkM3jud9wIOAbwC/VlXXDBJSmyTJ44E/qKqnJnkI3RGf+wLnA8+tqh8PmU/Lk2Q/ugHp2wFfB15A949k37dzLslrgV+n+xTt+cBv043Zafq9O5OFR5IkaSXN4iktSZKkFWXhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElq3v8HbftcJaFTR0cAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "State Values for learned policy:\n",
            "\n",
            "[[-5.21 -4.96 -4.74 -5.   -3.81  0.47  7.06 12.61 19.21 23.44]\n",
            " [-5.09 -4.65 -4.78 -4.62 -4.5   2.35  8.05 13.25 22.11 28.02]\n",
            " [-3.38 -4.95 -5.03 -4.62 -3.18  2.15  7.57 16.16 26.24 62.31]\n",
            " [ 9.39  4.95 11.27 -4.46 -1.9   3.1   9.76  0.   88.58 36.7 ]\n",
            " [ 5.13 18.32 20.99 26.52 41.29 57.93 82.64 84.65 80.04 25.47]\n",
            " [-3.22 13.7  23.79 27.72 33.82 38.71 83.09 14.71 81.95 32.73]\n",
            " [ 0.28 10.66 21.81 28.52 31.36 57.07 75.06  0.   27.32 10.36]] \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usynf6-9np24"
      },
      "source": [
        "## Epsilon Greedy Policy, Non-Deterministic Actions, Replacing Trace"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KfG9L8B1AYuR",
        "outputId": "ef3a75d1-1972-459e-e9a0-96b5b1e9012f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "Q = policyImprovement(deterministic=False, replaceTrace=True)\n",
        "print()\n",
        "plotPolicy(Q, deterministic=False)\n",
        "print()\n",
        "V = np.max(Q, axis=2)\n",
        "plotStateValues(V)\n",
        "print()\n",
        "print('State Values for learned policy:\\n')\n",
        "print(np.round(V,2),'\\n')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAGqCAYAAADHtHSPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAWDklEQVR4nO3dfYxlh3nX8d+dvbve9Xr8wq63dhOnjpMCThM8LaZE7qqZBCmElDolUKVSCaHlD0Ap3U6JVECg1VSqwCplO02DEBAlqOLFVBSivpBYJMwkkIbCWlPSOCF2HDtO/Fa79nps7/sc/pgZZzPZXa937pxzn3M/H8kazb27Ps+juXPnu+e+zKBpmgAAjLuprgcAALgUogUAKEG0AAAliBYAoATRAgCUIFoAgBKGF7tyamqq2bt3b1uztOb48eNJkj179nQ8yWjZq56+7mavevq6m71qev75559qmub6zZdfNFr27t2blZWV7ZuqI7Ozs0mSxcXFTucYNXvV09fd7FVPX3ezV02DweDh813u4SEAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBIGTdNc8MrhcNgcPHiwxXHasby8nCSZmZnpeJLRslc9fd3NXvX0dTd71bS0tHS0aZrbN1/uTAsAUMJFz7RMT083KysrLY7TjtnZ2STJ4uJip3OMmr3q6etu9qqnr7vZq6bBYOBMCwBQl2gBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChh0DTNBa8cDofNwYMHWxynHcvLy0mSmZmZjicZLXvV09fd7FVPX3ezV01LS0tHm6a5ffPlzrQAACVc9EzL9PR0s7Ky0uI47ZidnU2SLC4udjrHqNmrnr7uZq96+rqbvWoaDAbOtAAAdYkWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlDBomuaCVw6Hw+bgwYMtjtOO5eXlJMnMzEzHk4yWverp6272qqevu9mrpqWlpaNN09y++fLhxf7S2dXkc1/fvqG6cupkMhh0PQXAeHkuJ7OUh7oeY8ROZBB3+H1x0TMtU7unm5vuWmlxnHY88cHZJMmJBxY7nWPUZmdnkySLi4udzjFqfd0r6e9u9qpndnZ2LVgW/3rXo4zW7EeTJM3iQ52OMWp9vi0myWAwOO+ZFs9pAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChh2OnRmyY3Pnw8r/+D57L7xbNZuXZn7vvT1+a5fbs6HQsAGD+dRcv1j57InR/9WqafPZ2dp5oMkpzdkdxxzx/mkddfmd987005sbfbpgIAxkcnDw/tf/REfnzhwex78lR2rQdLkuw4mwzPNLnp/hfy3iMPZtfxs12MBwCMoU6i5c5/80h2nVy94PXDs8n0M6dz8L8+0eJUAMA4az1abvja8Vz9zKmX/XPDs03+1OeezfDUheMGAJgcrT9p5JYvPJfhqeaS/mwzlXznwy/ma9991TZPBcBWvOPnb82BL1/6ffXxa0/n1391eRsnoo9aj5bdx1cv+fROk2TnRR5GAqB71z18Zd45/4ZMrQ5e/g+vO3bjcdHCK9b6w0Mr1w5zZsel3bCnVpMXrvYKIoBxdvrKM3n8Dc91PQYToPVo+eL3XZtcYoyf3DOVx2/as70DAbAlz19/Kr/yyU/n0Tce63oUeq71aHn+2p158NarcmZ48XJpkhw9+MeSwaWfbgSgG88fOClc2HadvOT5d3781Xl2364Lhstq1k7GvOl/P5u9x063OhsAl0e4sN06iZZTu3fk1372lhz9wX05uXsqJ6+YyondUzm1ayrHrtuZxTu/I0/eeEX2PXkqP/ahh4QLQBEb4fLYG4QLo9fZs1xPX7EjS3fekP/xzgO54WvHs+vkal6YHubJV+1OBoN84fuvy3s+9NUceOxkfuxDD+U/vP/mvHDNzq7GBeASnbjmdG6875qux6CHOv8tz2eHU/nGLXvz1Vun8+Sr97z0HJbjVw1z9/tfu37G5aQzLgAFDE9O5Zd3v7vrMeipzqPlYoQLQB2bg+U//+L/zd974jc9x4WRGetoSYQLQAXnC5ZPfuDLnpzLSI19tCTCBWCcXShYNggXRqVEtCTCBWAcvVywbNgcLqs7Lu130MG5ykRLIlwAxsmlBsuGjXD5jV/6/dx19JNtjEjPlIqWRLgAjINXGiwbnj9wMp/62fvz/IGT2zkePVUuWhLhAtClyw0W2KqS0ZIIF4AuCBa6VDZaEuEC0CbBQtdKR0siXADaIFgYB+WjJREuANtJsDAuehEtiXAB2A6ChXHSm2hJhAvAKAkWxk2voiURLgCjIFgYR72LlkS4AGyFYGFc9TJaEuECcDkEC+Ost9GSCBeAV2R1IFgYa72OluT84bLjrN8uCnCuZjV5/Wf2v/S5YGEc9T5akm8PlwPfOCFcANadPbmaRz/zwkufCxbG1aBpLvzDezA1bK645WCL42yvqbNNDjx6Ik+cui/NYJA/8+bvzY5dg67HGpnl5eUkyczMTMeTjFZf90r6u5u96mhWk0c/80K+ni8lSXbfclOevenFjqcaoeXHkyRvmXlzx4OMVh9vi+daWlo62jTN7Zsvn4gzLRtWdwzy5HfuTjMYZNA0eXr5RM6ecsYFmEwbwbLhzO7VfgXLukH684/TSTe82JVX7d2TlQcWWxqlPT/4A2/JU79/Iu9/4cO57pkr8q5P3Zy9N+7seqwtm52dTZIsLi52Oseo9XWvpL+72Wv8nT25mn+x+76XPv/ILX8zV920sxe7natPX7Nz9XWvDYPB+UNzos60bJjaOcj+23Zn35uuyDNfOpmPve2hvPCYVxUBk2FzsNzxizfkqpvq/8ON/pvIaEnWwuVdn3qtcAEmyvmC5Xs/sP8ifwPGx8RGS5Ls2T8ULsDEECxUN9HRkggXYDIIFvpg4qMlES5AvwkW+kK0rBMuQB8JFvpEtJxDuAB9IljoG9GyiXAB+kCw0Eei5TyEC1CZYKGvRMsFCBegIsFCn4mWixAuQCWChb4TLS9DuAAVCBYmgWi5BMIFGGeChUkhWi6RcAHGkWBhkoiWV0C4AONEsDBpRMsrJFyAcSBYmESi5TIIF6BLgoVJJVouk3ABuiBYmGSiZQuEC9AmwcKkEy1bJFyANggWEC0jIVyA7SRYYI1oGRHhAmwHwQLfJFpGSLgAoyRY4FsNux6gbzbC5WNv+2qe/vxauNzxT2/IYMf2H/vEH51Nkjz88ZXtP1iSK28Y5vqZPa0cCyaNYIFvJ1q2weZw+e2/+HArx306J5Ikv/UX2jne1DB5+9035XXvvqaV48GkWD3T5N+/8YGXPhcssEa0bJONcFn624/m9MpqK8fc/Xtrp3Ne8/1XbfuxTq2s5vHPvph73vNI3n53hAuMyOqZJv/tfV/PsQdOJYPkbR9+VW79ieu6HgvGgmjZRnv2D/OOX39Na8f7pdndSZIf/vjN236spmnyub//RO696ynhAiOyESz3/7tj2XnVVH7ot74rr3rL3q7HgrHhibhclsFgkDf/4+/I9/3c/qyeSe55zyP5ym8c63osKGtzsPzwJ24WLLCJaOGyCRcYjfMFy413XNn1WDB2RAtbIlxgawQLXDrRwpYJF7g8ggVeGdHCSAgXeGUEC7xyooWRES5waQQLXB7RwkgJF7g4wQKXT7QwcsIFzk+wwNaIFraFcIFvJVhg60QL20a4wBrBAqMhWthWwoVJJ1hgdEQL2064MKkEC4yWaKEVwoVJI1hg9EQLrREuTArBAttDtNAq4ULfCRbYPqKF1gkX+kqwwPYSLXRCuNA3ggW2n2ihM8KFvhAs0A7RQqeEC9UJFmiPaKFzwoWqBAu0S7QwFoQL1QgWaJ9oYWwIF6oQLNAN0cJYOV+4HH/qTNdjwTc1ESzQkWHXA8BmG+GSJPfe9VSe+cLJ5Hs6HgqSpEme+dLJ3P9pwQJdGDRNc8Erh8Nhc/DgwRbHacfy8nKSZGZmpuNJRquPez334Kl88ZHPZ5DkTd9zW/bs71dn9/FrlvR0ryb5P797b86ebvKaqVuz77bd2XV1f05W9/JrFntVtbS0dLRpmts3X96f7zh66epbdmV4xVSaJM984aSHiujG+hmWs6ebDDLoXbBAFRc90zI9Pd2srKy0OE47ZmdnkySLi4udzjFqfd7ruQdP5W888i8zNUzefvdNed27r+l6rJHo89cs6cde5z7p9lemfjL7btudz9776a7HGrk+fc3OZa+aBoOBMy3UdfUtu7yqiNZtfpWQMyzQLd99lOHl0LTpfC9rFizQLd+BlOF9XGiL92GB8SRaKEW4sN0EC4wv0UI5woXtIlhgvIkWShIujJpggfEnWihLuDAqggVqEC2UJlzYKsECdYgWyhMuXC7BArWIFnpBuPBKCRaoR7TQG8KFSyVYoCbRQq8IF16OYIG6RAu9I1y4EMECtYkWekm4sJlggfpEC70lXNggWKAfRAu9JlwQLNAfooXeEy6TS7BAv4gWJoJwmTyCBfpHtDAxhMvkECzQT6KFiSJc+k+wQH+JFiaOcOkvwQL9JlqYSMKlfwQL9J9oYWIJl/4QLDAZRAsTTbjUJ1hgcgy7HgC6thEuSXLvXU/lEz/6SK647tFWjv3YsReTJB/e/8VWjteWNvdaPdPk1LFVwQITQLRAvj1cTjx9tpXjrqZJktaO15a29xIsMBlEC6zbCJeZD+xv7Zh3v2vth+xPfuxPtnbMNrS9144rBtk1vaOVYwHdES1wjsFgkD372/u2mNo5SJJWj9mGvu4FdMsTcQGAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCghEHTNBe8cjgcNgcPHmxxnHYsLy8nSWZmZjqeZLTsVU9fd7NXPX3dzV41LS0tHW2a5vbNlzvTAgCUcNEzLdPT083KykqL47RjdnY2SbK4uNjpHKNmr3r6upu96unrbvaqaTAYONMCANQlWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlCBaAIASRAsAUIJoAQBKEC0AQAmiBQAoQbQAACWIFgCgBNECAJQgWgCAEkQLAFCCaAEAShAtAEAJg6ZpLnzlYPCHSR5ubxwAgHxX0zTXb77wotECADAuPDwEAJQgWgCAEkQLAFCCaAEAShAtAEAJogUAKEG0AAAliBYAoATRAgCUIFoAgBJECwBQwrDrAS5kbn7h1Ul+Psk7kuxL8liS/5Jk/sjhQ890OdtW2KuWufmFv5LkLUlmktyWZDrJvz1y+NBf7XSwLbJXLXPzC/uS/KUkP5TkTUleleRUks8n+UiSjxw5fGi1uwkvT1/3Svp7W9zQ1X3+WJ5pmZtfeF2So0l+IsnvJTmS5MEkh5L87voNvRx7lfQPk/xU1u54vtHxLKNkr1p+NMm/SvJnk/yvJL+c5D8leWOSf53kP87NLwy6G++y9XWvpL+3xU7v88f1TMs/T3IgyU8fOXzogxsXzs0v/LMkc0l+Icnf6mi2rbBXPXNJvp7kgaz9q+m/dzvOyNirli8nuTPJb5975mFufuEfZO2Hxl9O8u6s/cCvpK97Jf29LSYd3ucPmqbZjv/vZVsvuAeSPJTkdZtuyNNZOwU1SHLgyOFDL3Qy5GWwV629zmdufmE2a3c8vTnFm9iruvUf8L+Q5FePHD70d7qeZ1T6tFefbotd3+eP48NDb13/eM/mxzKPHD60kuR/JrkyyZvbHmyL7AVsh9PrH890OsXo9XWv6jq9zx/HaPkT6x+/fIHr71//+MdbmGWU7AWM1Nz8wjDJX1v/9ONdzjJKfd2rJzq9zx/HaLlm/eOxC1y/cfm1LcwySvYCRu2fZO1Jq79z5PChT3Q9zAj1da8+6PQ+fxyjBYCXMTe/8NNJ/m6SLyV5b8fjjExf92I0xjFaNirtmgtcv3H5sy3MMkr2AkZibn7hp5IsJLkvyVuPHD70Rx2PNBJ93atnOr3PH8do+X/rHy/0eNh3r3+80ONp48pewJbNzS/8TJIPJvmDrP1gf7zjkUair3v1UKf3+eMYLRuvZX/73PzCt8y3/nKqH0jyYpLPtT3YFtkL2JK5+YWfy9obeS1n7Qf7kx2PNBJ93aunOr3PH7toOXL40FeS3JPk5iTv33T1fJK9SX6t2nt+2KvWXjBu5uYX/lHWnqB6NMmfO3L40FMdjzQSfd2rr7q+zx+7N5dLXnrzms9m7R33Ppbki1l7m+e3Zu2U0x1HDh96ursJL4+96pmbX/iRJD+y/ukNSf581t6u+jPrlz115PChD3Qx21bYq5a5+YX3JflokrNZewjlfK/ceOjI4UMfbXGsLevrXkl/b4tJt/f5Y/k2/kcOH/rK3PzC7fnmL2N6Z9beZW8hhX8Bn71Kmknyvk2X3bL+X5I8nKTiHY+9annt+scdSX7mAn9mKWsBUElf90r6e1vs9D5/LM+0AABsNnbPaQEAOB/RAgCUIFoAgBJECwBQgmgBAEoQLQBACaIFAChBtAAAJYgWAKAE0QIAlPD/AX9apZ+cplb1AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAGSCAYAAADn8RuZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5gkdX3v8fdHFhDQgIgSIkY0EpR4IuDGo5EQATF4wdsxCYniJUaMUSNJFFFzTvDkMVETjfqY6MMRBQWvKN4SFVQMMVF0l4tyEa+oeGFFFBAvKHzPH1UjQ9Pd2zM7u9Xz2/frefqpnurqmk/9umbms1XVvakqJEmSWnaroQNIkiRtbhYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIjUry8SSXDZ1jFkl2TPLqJF9PcsNqyb2pkhyfpJLsNXSWSZKclKRG5s19bmmUhUerSpK7JTkhyeeT/CjJ95NckuTkJAePLHt8kketwPc8JsmTNnU9Y9b7sP6Pxis3styL+uX+eKUzzJHnAc8C3g48CThm0DSrUF8aj0nyn0muSvKzJFck+fckT0qyZuiM0pD8AdCqkWQt8B/Az4A3ARcBOwB7Aw8GrgXOWvSUvwVOBt6zid/6GOAy4KRNXM+oDwHfAh6X5LlV9bPRBZIEeCLwA+DdK/z958lhwOeq6rlDB1mNktwd+Dfg14GPAP8AXAncEXgQ8EZgX+DYoTJKQ7PwaDX5W2BHYL+qumD0wSS/vOUjLV9V3ZDkJOAFwBGMLzSHAHcB/rWqfrIF421pvwx8faVXmuS2VXXtSq93niTZAfgAcDfgf1XV6H700iS/BfzWFg8nzRFPaWk12Rv43riyA1BV3wFIsteiaw6e2J8OqsXXIST5wyTv668Z+WmSK5O8J8lvLl5n/5y7AL+7eD2Lr11IsjbJ6f06fprk0iQvnPEUwhuAAv5kwuN/smi5mXNPkuSyJB8fM/+B/XY9aWT+9klekOSiJD9J8oMk70+y/8hyt+pPp3w2ybVJrunH4cQk207J86R+jO/Kzcf4+EXLPCrJfyW5LskP+/uPnLRtSfZP8uEkVwOfnWFM9kjy2n5Mr0/yrf606R1HlvuVJC9Pcn5/KvUnSS5O8rwk24xZ73ZJju2X/1GSq5OsS/LMMTG2T/L3SS7vX9cLkjx0Y9l7fwrsA7x8TNkBoKo+U1X/OpJvU/ZbadVxx9Zq8mVgnySPmfSLvfdd4CjgzcB/AieMWeaZwPf6x74D/BpwNPBfSQ6oqi/2yx0F/DPd6YEXj3wPkjyM7sjMl4CXA1cB9wf+L7Af8PvTNqiqvpzkbODwJHtU1bcXHkuyM/Bo4IKqWr/E3JusLyofAn6bbixfA+wMPLX/fgdV1bp+8RfSbfP7gdcBN9CVmEcA29OdhhznbMaP8Wf7DH8O/Avw+X790F3j854kT6uq0df2V4GPAe8E3gXcZiPb+KvAJ4HtgBPp9rG7A08HDk6ytqqu7hf/TeAxwOn9ctsChwMvoTu68rRF690O+DDwQOAM4BTgJ8D/6NfxmpEoJ9ON0T/1WY7pt/HXq+qyadsAPLafjtvPJ233Ju230qpUVd68rYob3S/k6+mOiHyB7qjH04F7Tli+gJMmPLbTmHn3BH5Kd/po8fzLgI+PWf7WdKXjbGDNyGN/2X//B86wXUf1yx47Mv9p/fy/WGbujwOXzbgtD+y/15PGbMPvjSz7S3Snnz6+aN65wMWb8NreIhdwO+CHdH+Uf2nk+3+Z7pqtXUbWUcCfLuH7vhfYAOw5Mn8t8HPg+EXzdgAyZh1vpit4eyyad2yf5e/HLH+rRfeP75f7wOJ1051+KuAfZtiG7wFXL2Gbl7Tf0l27ViPLLeTea7mvuTdvW/rmKS2tGlX1SeA+dP8a3hl4MvCvwMVJzk5ytyWs6zroLgpO8ktJdqM7anMp8D9nXM1hwO50F4TukmS3hRvw7/0yD55hPacBV/fbs9iT6YrMKSuce1aPpzuysn5k27YDzgQOTHf9CH3+OyU5cAW//2HATsCrq+qahZn9/VfTHb150MhzrqJ7PTaqP4L2cOB9wE9GtvEyuqL1i9evqn5cVdU/d7sku/bLfpju8oC1i1b/OOD73HRUikXruXFMnFctrLtf5jN0ZW/vGTbll+jK36xWar+VVhVPaWlVqarP0Z3SIMldgN+lu4bhd4D3JrlPVV2/sfX016D8Hd2RjZ1GHv7qjHHu2U/fMGWZ3Te2kqr6cZK3An+W5P5V9ckk+9IVmHdU1VUrnHtW96Q7qvHdKcvsBnyD7sLr9wD/meRbdEeX/g04bZbXY4K79tOLxjy2MG+05H65qm6Ycf370BWVp/S3cb6ycKe/tuU44Al0p70ysuztFt3fGzi/Zr/Q/Ctj5n0PuP0Mz70GuO2M3wdWaL+VVhsLj1atqvoa8KYkC9fqPAC4L/CJac/rr9s4m+4Pxd/RHR25ju4Q/SvZyHUfi1fVT58LnD9hmW/NuK43AH9Gd1Tnk9x0tOfEFc5dE+aP+10Q4HPAX01Z33ehO/qW5NeA3wMO7m9/DPxNkgMXl7bN7EdLWHbh9TuF7qjhOD9edP8V3PRZQS+mOxX2M+AA4KVs2ptAJpW00VI1zoXAQUnuVlXjitOkda7EfiutGhYerXpVVUnOoSs8d5rhKY+mKwePqKrFn9tDktvTnUa62beYsJ6FC4Svq6qPLCHyLVTVZ5J8DvjDJM+hu67n63SfqbLc3ONcBew6Zv6404FfBO4AfGzCaZjRbfgh3YXC7+ozLVxw/BTgH2fINmrhj/dvAB8deWzfkWWW40t0r+12M75+RwFnV9WRi2em+wycUV8A7pFk+6qa5XXZFO8CDqI70vmCGZZfsf1WWk28hkerRpLDxr1ltr+OZOGag4sXPfRDxv9xX/jX9M3+9ZzkqXSfBzNq0no+TPev/OOS3OLxJDskWcqphhPprsd4Pd0phZNGisZSc4+z8If4F8UwyfbAM8Ys+6Z+vWOP8CTZfdH93cYscm4/HTd2sziT7gjWsxaPY3//WXSvy5nLXDdV9T26a1Yek+R+o4/310ndYdGsG7jl2O9Ed6HvqFPpTnH9zbj1LjfzBK+nO9r3nIx5u37/Pe/TF1BY+f1WWhU8wqPV5J+B2yd5H92plh8Bd6Y7dfLrwJv6a3wWfAp4UJLn0R0tqap6G/DB/rlvTvIauotLHwA8lO7dP6M/F58CnpLk74BLgBuB91fVdUmeQHftyqVJ3kB31GAX4B50bz9+NN31LLM4BXgZ3VuCi1tefLvU3OO8BjgS+EiS19FdgHwU408FvYruAtd/THII3du9r6F76/ehdG+zXvjvPC5J8ingHLrTIXvQvV3+euBtM+S6har6QZJj6Y4SnZPuQxqhu4br7sDT6qa3jC/X0+lOgZ6d5E3AeXT/ELwb8Ei60nd8v+xpwNOSvJ3uyNvudJ+T9L0x630V3YdJ/k26D/07g268foPu2qHRi62Xrap+lOThdNdMvSfJGXRF8Ht0R+gOpjvV+LJ++ZXeb6XVYei3iXnzNuuN7ijOvwAX0H1my8/pfqmfRfeH51Yjy+9N94fmGroCUYseO4juD921dP9tw78B92L8W7nvSHfa4Cq6snOzt+P2zzsF+CbdH/grgP8G/jew6xK38e39+j864fGl5L7FvH7+E+mOCFxPd6HzsXSf6Hyzt6X3y64B/gL4DN3RluvoTomcCjx40XLH0V1ftIHu1No36D4L54AZt/syxrxdvn/s0f14Lnz//wYetZR1bOR770Z3yu0LdKXkB3SF+lXAvouW27Ff7mv9cl/st/vQCWN3a7rPJ7po0Xo/A/z5omWOH92flrs9fb6/7PeP79NdX3RFv48cBWwzsvxM+y2+Ld1bI7dUTbo8QZIkqQ1ewyNJkppn4ZEkSc2z8EiSpOZZeCRJUvN8W/omSnIE3dtPnzp0FklSk66sqjtsfDFN47u0VkiSmuexXPiss6ql/B+DW9bCZ511/2/ifOo+UgWqTtnIksNIHg9A1aMHTjJZcjoAVXccOMlkyQYA6sUDB5kiL+ymda9hc0yTC7vpWdMXG8zCh0jtM2iK6S7tJuurau30JbUxntKSJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhmSDJG5JsSHLh0FkkSdKmsfBMdhJw+NAhJEnSprPwTFBVZwNXDZ1DkiRtujVDB1jNkhwNHD10DkmSNJ2FZxNU1QnACQBJauA4kiRpAk9pSZKk5ll4JElS8yw8EyR5K/BJYJ8klyd5ytCZJEnS8ngNzwRV9UdDZ5AkSSvDIzySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmpeqGjpDE5I4kJKkzWF9Va0dOsRqt2boAKtdkiOAI4bOIUmSJvMIzwpJUvM8lkkAqLpk4CSTJfcEVkvG8wZOMl6yPwBVHxk4yWTJgwCoes3ASSZLnglAfW3gIFPkLt20Xjxsjmnywm5auw6bY5Jc1U3vPWyMqS7oJh7hWQFewyNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVngiR3TnJWkouTXJTk2UNnkiRJy7Nm6ABz7OfAX1fVuUluC6xPcmZVXTx0MEmStDQe4Zmgqr5dVef2968FLgHuNGwqSZK0HBaeGSTZC9gfOGfYJJIkaTk8pbURSW4DvAs4pqquGXnsaODoQYJJkqSZWXimSLItXdk5tarePfp4VZ0AnNAvW1s4niRJmpGntCZIEuBE4JKqesXQeSRJ0vJZeCZ7AHAUcEiS8/vbQ4cOJUmSls5TWhNU1SeADJ1DkiRtOo/wSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDUvVTV0hiYkcSAlSZvD+qpaO3SI1W7N0AFWuyRHAEcMnUOSJE3mEZ4VkqSqrh06xkTJbQGoOmvgJJMlBwNQdcnASSZL7glA1VcHTjJeclcAql43cJLJkj8DoOqOAyeZLNkAQF04cJApcq9uWo8bNsc0ObWbXj1sjIl27qc7Dppiuh91E4/wrACv4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IzQZJbJ/l0kguSXJTkRUNnkiRJy7Nm6ABz7KfAIVX1wyTbAp9I8sGq+tTQwSRJ0tJYeCaoqgJ+2H+5bX+r4RJJkqTl8pTWFEm2SXI+sAE4s6rOGXn86CTrkqwbJqEkSZqFhWeKqrqhqvYD9gTum+ReI4+fUFVrq2rtMAklSdIsLDwzqKofAGcBhw+dRZIkLZ2FZ4Ikd0iyS39/B+Aw4PPDppIkScvhRcuT7QGcnGQbumL4jqr6wMCZJEnSMlh4JqiqzwL7D51DkiRtOk9pSZKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDUvVTV0hiYkcSAlSZvD+qpaO3SI1W7N0AFWuyRHAEcMnUOSJE3mEZ4VkqTqwKFTTJZPdNO6Ytgc02T3blr1/GGDTJH8AwBV/zRwkvGS5wBQdd+Bk0yWfBqAunrgIFNk525anxg2xzTpf99U/XjYIFMkOwAwr8cmsq6bHjxsjKnO6iYe4VkBXsMjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FZyOSbJPkvCQfGDqLJElaHgvPxj0buGToEJIkafksPFMk2RN4GPD6obNIkqTls/BM90rgWODGcQ8mOTrJuiTrtmwsSZK0FBaeCZI8HNhQVesnLVNVJ1TV2qpauwWjSZKkJbLwTPYA4BFJLgPeBhyS5JRhI0mSpOWw8ExQVc+vqj2rai/gSOBjVfX4gWNJkqRlsPBIkqTmrRk6wGpQVR8HPj5wDEmStEwe4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqXqpq6AxNSOJASpI2h/VVtXboEKvdmqEDrHZJjgCOGDqHJEmazCM8KyRJVe07dIyJkosBuGrgHNPs2k/rTYPGmCpP6KZVBwwbZILkXADqEwMHmSIHdtP61rA5psmvdNN62bA5psmx3bSeO2yOafKP3bR2HjbHJLm6mz552BhTvbGbeIRnBXgNjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5a4YOMM+SXAZcC9wA/Lyq1g6bSJIkLYeFZ+MOrqorhw4hSZKWz1NakiSpeRae6Qo4I8n6JEePPpjk6CTrkqwbIJskSZqRp7SmO7CqvpnkjsCZST5fVWcvPFhVJwAnACSpoUJKkqTpPMIzRVV9s59uAE4H7jtsIkmStBwWngmS7JTktgv3gQcDFw6bSpIkLYentCbbHTg9CXTj9Jaq+tCwkSRJ0nJYeCaoqq8A9x46hyRJ2nSe0pIkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUvFTV0BmakMSBlCRtDuurau3QIVa7NUMHWO2SHAEcMXQOSZI0mUd4VkiSqvrk0DEmSu4PQJ01cJApcnA3rScMm2OavKmb1rphc0yS/t+AddqwOabJY7tpvW/YHNPkEd203jFsjmnyB920jhs2xzR5STetRw+bY5Kc3k33HDbGVJd3E4/wrACv4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IzRZJdkpyW5PNJLkly/6EzSZKkpVszdIA59yrgQ1X12CTbATsOHUiSJC2dhWeCJDsDBwFPAqiq64Hrh8wkSZKWx1Nak90V+C7wxiTnJXl9kp2GDiVJkpbOwjPZGuAA4LVVtT9wHXDc4gWSHJ1kXZJ1QwSUJEmzsfBMdjlweVWd0399Gl0B+oWqOqGq1lbV2i2eTpIkzczCM0FVfQf4RpJ9+lmHAhcPGEmSJC2TFy1P9yzg1P4dWl8BnjxwHkmStAwWnimq6nzA01WSJK1yntKSJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJal6qaugMTUjiQEqSNof1VbV26BCr3ZqhA6x2SY4Ajhg6hyRJmswjPCskSVV9d+gYEyV3AKDq9wdOMlnyTgDqtIGDTJHHdtNPDBtjogP7ab1x0BhT5cndtB43bI5pcmo3rROGzTFNju6mdeKwOabJU7rpB4eNMdFD+ulLBk0x3XHdxCM8K8BreCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BMkGSfJOcvul2T5Jihc0mSpKVbM3SAeVVVlwL7ASTZBvgmcPqgoSRJ0rJ4hGc2hwJfrqqvDR1EkiQtnUd4ZnMk8NbRmUmOBo7e8nEkSdJSeIRnI5JsBzwCeOfoY1V1QlWtraq1Wz6ZJEmalYVn4x4CnFtVVwwdRJIkLY+FZ+P+iDGnsyRJ0uph4ZkiyU7AYcC7h84iSZKWz4uWp6iq64DbD51DkiRtGo/wSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDUvVTV0hiYkcSAlSZvD+qpaO3SI1W7N0AFWuyRHAEcAPwIuWeHV7wZcuYLr2xm4egXXB/OfcaXzwfxn3BpfZ5j/jFvjvgjzn3E17It7r+C6tl5V5W0FbsAJm2Gd68w4X/lWQ8at8XVeDRm3xn1xNWTcGvfFrfXmNTwr5/1DB5iBGVfGvGec93xgxpVixk037/lgdWScexaeFVJVc79DmnFlzHvGec8HZlwpZtx0854PVkfG1cDCM99OGDrADOY947znAzOulHnPOO/5wIwrZTVk3Or4Li1JktQ8j/BIkqTmWXgkSVLzLDxzKMkbkmxIcuHQWcZJcuckZyW5OMlFSZ49dKZRSW6d5NNJLugzvmjoTOMk2SbJeUk+MHSWcZJcluRzSc5Psm7oPOMk2SXJaUk+n+SSJPcfOtNiSfbpx2/hdk2SY4bONSrJX/Y/KxcmeWuSWw+dabEkz+6zXTQv4zfud3WSXZOcmeSL/fR2Q2bUTSw88+kk4PChQ0zxc+Cvq2pf4H7AM5LsO3CmUT8FDqmqewP7AYcnud/AmcZ5Niv/gZUr7eCq2q/m95NeXwV8qKruAdybORvPqrq0H7/9gPvQfUjp6QPHupkkdwL+AlhbVfcCtgGOHDbVTZLcC3gqcF+61/jhSe4+bCpg/O/q44CPVtXewEf7rzUHLDxzqKrOBq4aOsckVfXtqjq3v38t3R+YOw2b6uaq88P+y23721xdoZ9kT+BhwOuHzrJaJdkZOAg4EaCqrq+qHwybaqpDgS9X1deGDjLGGmCHJGuAHYFvDZxnsXsC51TVj6rq58B/AI8ZONOk39WPBE7u758MPGqLhtJEFh5tkiR7AfsD5wyb5Jb600XnAxuAM6tq3jK+EjgWuHHoIFMUcEaS9UmOHjrMGHcFvgu8sT81+PokOw0daoojgbcOHWJUVX0T+Cfg68C3gaur6oxhU93MhcDvJLl9kh2BhwJ3HjjTJLtX1bf7+98Bdh8yjG5i4dGyJbkN8C7gmKq6Zug8o6rqhv40wp7AffvD4nMhycOBDVW1fugsG3FgVR0APITu1OVBQwcasQY4AHhtVe0PXMecnkJIsh3wCOCdQ2cZ1V9n8ki6AvkrwE5JHj9sqptU1SXAS4EzgA8B5wM3DBpqBtV97stcHVnemll4tCxJtqUrO6dW1buHzjNNf4rjLObruqgHAI9IchnwNuCQJKcMG+mW+n/5U1Ub6K47ue+wiW7hcuDyRUfvTqMrQPPoIcC5VXXF0EHGeBDw1ar6blX9DHg38NsDZ7qZqjqxqu5TVQcB3we+MHSmCa5IsgdAP90wcB71LDxasiShu2bikqp6xdB5xklyhyS79Pd3AA4DPj9sqptU1fOras+q2ovuNMfHqmpu/kUNkGSnJLdduA88mO7Uwtyoqu8A30iyTz/rUODiASNN80fM4ems3teB+yXZsf/5PpQ5u/g7yR376a/SXb/zlmETTfQ+4In9/ScC7x0wixZZM3QA3VKStwIPBHZLcjnwt1V14rCpbuYBwFHA5/prZABeUFX/PmCmUXsAJyfZhq7Yv6Oq5vKt33Nsd+D07u8fa4C3VNWHho001rOAU/tTRl8BnjxwnlvoC+NhwNOGzjJOVZ2T5DTgXLp3YZ7H/P33CO9KcnvgZ8Az5uHi9HG/q4GXAO9I8hTga8AfDJdQi/lfS0iSpOZ5SkuSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLz1kx78PDDD68rr7wSqJFHamQ6bt5SnjPLOmZddnS5eV920uNLec5KrX/Wdc26zIzfZtq8TXnO5l7/Sj1nS2caapuXst4hMg29zRubt8TVb+74re5GS1n/LN9zHn4NzUOmLbXNN8KHq+rwMQ9NLzxXXnkl69atA37ez5k0nfbYwvSGZTxnOcvO8pxNybLS+UezDDk+k7Is4fvUDTcteuPIam+YMP9GbrKxZRetfuJ6Jq1jc2UZ95xZt32WLMtZdto4TVv/rMsu53VYyvq31OuwlCyzvA6baZ+Yx12u1d1oKVm2hl8tK/E6LyXLpv7qug52YwJPaUmSpOZZeCRJUvMsPJIkqXkWHkmS1DwLjyRJap6FR5IkNc/CI0mSmmfhkSRJzbPwSJKk5ll4JElS8yw8kiSpeRYeSZLUPAuPJElqnoVHkiQ1z8IjSZKaZ+GRJEnNs/BIkqTmWXgkSVLzLDySJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOZZeCRJUvNSVZMfTD4E7DYyezfgys0ZapVwHByDBY5Dx3HoOA4dx6HjOHS21DhcWVWHj3tgauEZ+4RkXVWtXZFYq5jj4BgscBw6jkPHceg4Dh3HoTMP4+ApLUmS1DwLjyRJat5yCs8JK55idXIcHIMFjkPHceg4Dh3HoeM4dAYfhyVfwyNJkrTaeEpLkiQ1z8IjSZKaN3PhSXJ4kkuTfCnJcZsz1LxJclmSzyU5P8m6ft6uSc5M8sV+eruhc660JG9IsiHJhYvmjd3udF7d7x+fTXLAcMlX1oRxOD7JN/t94vwkD1302PP7cbg0ye8Nkwyg074AAAS6SURBVHrlJblzkrOSXJzkoiTP7udvNfvElDHYqvaHJLdO8ukkF/Tj8KJ+/l2TnNNv79uTbNfP377/+kv943sNmX+lTBmHk5J8ddH+sF8/v7mficWSbJPkvCQf6L+er/2hqjZ6A7YBvgzcDdgOuADYd5bntnADLgN2G5n3MuC4/v5xwEuHzrkZtvsg4ADgwo1tN/BQ4INAgPsB5wydfzOPw/HAc8Ysu2//87E9cNf+52abobdhhcZhD+CA/v5tgS/027vV7BNTxmCr2h/61/Q2/f1tgXP61/gdwJH9/NcBT+/v/znwuv7+kcDbh96GzTwOJwGPHbN8cz8TI9v3V8BbgA/0X8/V/jDrEZ77Al+qqq9U1fXA24BHzvjcVj0SOLm/fzLwqAGzbBZVdTZw1cjsSdv9SOBN1fkUsEuSPbZM0s1rwjhM8kjgbVX106r6KvAlup+fVa+qvl1V5/b3rwUuAe7EVrRPTBmDSZrcH/rX9If9l9v2twIOAU7r54/uCwv7yGnAoUmyheJuNlPGYZLmfiYWJNkTeBjw+v7rMGf7w6yF507ANxZ9fTnTf8hbU8AZSdYnObqft3tVfbu//x1g92GibXGTtntr3Eee2R+WfsOiU5pbxTj0h6D3p/sX7Va5T4yMAWxl+0N/+uJ8YANwJt3Rqx9U1c/7RRZv6y/GoX/8auD2Wzbx5jE6DlW1sD+8uN8f/jnJ9v28ZvcH4JXAscCN/de3Z872By9ans2BVXUA8BDgGUkOWvxgdcfltrr392+t2917LfBrwH7At4GXDxtny0lyG+BdwDFVdc3ix7aWfWLMGGx1+0NV3VBV+wF70h21usfAkQYxOg5J7gU8n248fgvYFXjegBE3uyQPBzZU1fqhs0wza+H5JnDnRV/v2c/bKlTVN/vpBuB0uh/uKxYORfbTDcMl3KImbfdWtY9U1RX9L7obgf/HTacpmh6HJNvS/aE/tare3c/eqvaJcWOwte4PAFX1A+As4P50p2jW9A8t3tZfjEP/+M7A97Zw1M1q0Tgc3p/6rKr6KfBG2t8fHgA8IslldJe8HAK8ijnbH2YtPJ8B9u6vuN6O7iKj922+WPMjyU5JbrtwH3gwcCHd9j+xX+yJwHuHSbjFTdru9wFP6N+FcD/g6kWnOZozct790XT7BHTjcGT/LoS7AnsDn97S+TaH/hz7icAlVfWKRQ9tNfvEpDHY2vaHJHdIskt/fwfgMLrrmc4CHtsvNrovLOwjjwU+1h8NXNUmjMPnF/0DIHTXrSzeH5r6mQCoqudX1Z5VtRddP/hYVT2OedsfZr26me7q8i/Qnad94Za4onoebnTvTLugv120sO105xs/CnwR+Aiw69BZN8O2v5Xu8PzP6M6/PmXSdtO96+Bf+v3jc8DaofNv5nF4c7+dn6X74d1j0fIv7MfhUuAhQ+dfwXE4kO501WeB8/vbQ7emfWLKGGxV+wPwm8B5/fZeCPyffv7d6Ardl4B3Atv382/df/2l/vG7Db0Nm3kcPtbvDxcCp3DTO7ma+5kYMyYP5KZ3ac3V/uB/LSFJkprnRcuSJKl5Fh5JktQ8C48kSWqehUeSJDXPwiNJkppn4ZEkSc2z8EiSpOb9f2vYdyaYfvi4AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "State Values for learned policy:\n",
            "\n",
            "[[ -3.36   3.21  19.62  36.08  70.74  94.82 180.52 236.9  305.19 401.  ]\n",
            " [ -2.96  14.07  13.69  17.04  25.69  42.5  122.95 181.59 254.24 396.71]\n",
            " [  3.05  24.68  13.87  11.25  40.56  94.5  132.23 207.29 274.22 420.07]\n",
            " [227.46 118.87  53.6   45.48  82.42 113.18 141.1    0.   232.6  384.15]\n",
            " [ 86.69 275.4  162.58  83.76 141.87 119.65 178.14 190.86 255.58 365.29]\n",
            " [ 21.07 144.5  206.23 137.09 156.19 153.04 157.35 189.28 214.58 408.08]\n",
            " [  5.6   68.42 156.38 301.18 164.41 206.83 167.66 166.89 310.19 340.61]] \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}