Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "from collections import deque\n",
    "import torch\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "# environment configuration\n",
    "env = UnityEnvironment(file_name='Reacher.app')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Number of actions: 4\n",
      "States look like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n",
      "States have length: 33\n"
     ]
    }
   ],
   "source": [
    "# environment information\n",
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "n_agents = len(env_info.agents)\n",
    "print('Number of agents:', n_agents)\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ddpg_train(agent):\n",
    "    scores = []\n",
    "    scores_window = deque(maxlen=100)\n",
    "    n_episodes = 1000\n",
    "\n",
    "    for episode in range(n_episodes):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]            # reset the environment\n",
    "        states = env_info.vector_observations\n",
    "        agent.reset()                                                # reset the agent noise\n",
    "        score = np.zeros(n_agents)\n",
    "        \n",
    "        while True:\n",
    "            actions = agent.act(states)\n",
    "\n",
    "            env_info = env.step( actions )[brain_name]               # send the action to the environment                            \n",
    "            next_states = env_info.vector_observations               # get the next state        \n",
    "            rewards = env_info.rewards                               # get the reward        \n",
    "            dones = env_info.local_done                              # see if episode has finished        \n",
    "\n",
    "            agent.step(states, actions, rewards, next_states, dones) # update the score\n",
    "\n",
    "            score += rewards                                         \n",
    "            states = next_states                                     # roll over the state to next time step        \n",
    "                                                        \n",
    "            if np.any( dones ):                                      # exit loop if episode finished        \n",
    "                break                                        \n",
    "\n",
    "        scores.append(np.mean(score))\n",
    "        scores_window.append(np.mean(score))\n",
    "\n",
    "        print('\\rEpisode: \\t{} \\tScore: \\t{:.2f} \\tAverage Score: \\t{:.2f}'.format(episode, np.mean(score), np.mean(scores_window)), end=\"\")  \n",
    "        \n",
    "        if np.mean(scores_window) >= 30.0:\n",
    "            # Save model weights\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            \n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(episode, np.mean(scores_window)))\n",
    "            break    \n",
    "\n",
    "    plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: \t109 \tScore: \t34.19 \tAverage Score: \t30.20\n",
      "Environment solved in 109 episodes!\tAverage Score: 30.20\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvs0lEQVR4nO3dd3yUVbrA8d+T3nshpFBC6J0ISFEQUER3bbsq9rWwugV116vurnuvu3v3rttw1et1RUWxYcWGFREElZZAgACB0ALpCaQRkpBkzv1jXjBAAoFkMpmZ5/v5zCczZ2byPm9eeObM8573HDHGoJRSynN4OTsApZRSXUsTv1JKeRhN/Eop5WE08SullIfRxK+UUh7Gx9kBtEdMTIzp3bu3s8NQSimXkpmZWW6MiT253SUSf+/evcnIyHB2GEop5VJEJK+1di31KKWUh9HEr5RSHkYTv1JKeRhN/Eop5WE08SullIfRxK+UUh5GE79SSnkYTfyq21q2vYTckhpnh+Ewm/MrqTva7OwwlAfSxK+6VG1DE+1ZA2LFjlLuWJjBTS+s5VDt0S6IrGuVVNdz5dPf8q9lO50diupCzTbDrtLDzg5DE7/qOuWHGxj75y+Zt/T0ya6oqo5fvbWJPjHBVNQ28tC7m0/4sKhtaHJ0qA63Krccm4H3NxbQbNPFkDzFy6v3MePxr8kprnZqHJr4VZf5IKuQ2qPN/O/yXazeffB4e3FVPe9tzCenuJqjTTbmLtpIfWMzz9+azoMzB7B0Wwmvrd1PdkEVty5Yx7BHP+elb/c6cU/a58jRJj7LLuaBtzfx189yTnhuVW4ZACXVDXy3u7zdv7Ox2caTy3LJLqjq1FjVmTXbDEs2F3ao4/FOZj7GwKK1+09of+T9LTy9fFdHQ2w3l5irR7mHdzPzGZQQRn1jM796K4vP7r2AbUXV/OL1DRy0yjl+Pl4cbbLxxPUjSY0NoU90MKtyy3n0w6002Qzhgb6MTonk0Y+2caj2KPfP6I+ItLq9uqPNPL9qD9ePTSE21L8rd5VPtxRx35tZNDTZ8PYSmm2GG8elkBQZhM1mWJVbzmXDEliZW8biDQVMTjtlHq1WzV+5h3lLd/LMit08NXsU0wfHn1N8zTaDt1frfzfVun98sYNnVuzmtgm9efSHQ876/bklNWwtrCbU34f3Nhbw8KWDCPTzJutAJa+u2U94oC93Te6Ln4/j++Pa41ddYntRNduKqrn+vGSeuH4kZTUNXP/cGm56YS0RQb68MWc8//zxCH40JokHZw7gipGJAHh5Cf/48QhGp0TysymprHxwKm/MGc916ck8+dUuHnh7MwWVda1u8/cfZPPPpTt5/ps9XbmrlFbX8/DiLfSPD+X1u8ax7FcXAvayDsC2omoO1R5l+uA4Lh/ek8+yi9vVi9xVepgnluVy0cA4+sWFMOeVDF48h28+L327l4G//5R7Xs3ku93l7TrncraONtl4evkudnaTk/NNzTb++NE2vtha3K7XZ+ZVsDyn9Pjf5tMtRTyzYjeRQb68vnY/+RVHWn3f4YYmqusbW33u/awCvL2Ev1wzjOr6Jj7ZUgTAk8ty8RKoqmvk27P49tcR2uNXXeLdzHx8vYUfjuhJZLAfv7q4P3/7bAczBscz79oRhAb4AnDNmKRT3hsb6s9bd59/Qttj1wwjLsyfp5fv4r2N+Vw8uAd3TO7Deb2jAHhr/QHeycwn2M+bjzcX8fDMgW1+M+hMxhh++94W6hubeeL6kfSNDQFgXJ8oFm8o4OdT+7HSKvNM7BdDcmQQi9bt57PsYq4Zk8Sh2qN8sqWILflVZBdWERbgywOX9GdkciQPvbuZID9v/nrNcIL9vbn3jSz+8NE2Qvx9+HF6crvieyczn0c/2saIpHDW7DnIp9nFDIgP5f4ZaVwypAci9m8n2QVVxIT6kxgReE5/h69ySvn75zt4fOlO7pjUh7nT0gj2d166eezTHBZ8u5flO0qZMTj+tP8Wvt5Zxl0LMzjabGNMr0huGp/CI+9lMyolgsevHcnFj6/kyWW5/O1HI054n81muO7Z1eSWHmbW0B5cPzaFcX2iEBFsNsP7GwuZ1C+Gy4YlMC9mJ4vW7ad/fChf5ZQyd1oaL367lyWbipg6IM7Rfw5N/MrxGpttvJ9VwEUD44gM9gPgngtTuSAtlsEJYXidQ8lBRPj1xQO47rxkXlmTx5vrD/DZ1mIm9ovmqlFJ/P6DbCb1i+GHI3vy4DubyTpQyaiUyM7etVMs3lDAl9tLeeSyQceTPsA1o5N48F17HKt2ljMoIYy40ABiQ/xJiQri7cwDHG5o4p9f7KC6vonIIF+GJoazs6SGa55ZzdDEMLILqpl37YjjZat/3zSGm55fy+8/yGZEcgT940NPG9tn2cU8+M4mJvWL4YXb0jEGPtpUyL+/3s3dr25gaGIYaXGhrNhRSsURe6+1f3wIE1JjqG1oIu/gEarrG5k6MI4rRvZkYI+wNre1enc5gb7e/GBEAs+u3MPHW4r48BeTiLKOf1d6NzOf57/ZS//4EHaWHGbD/krG9Gr938K3u8qZ83IG/eJCmD0uhaeW5XL/m5uICfHn3zeNIT4sgBvHp/Dy6jzuvjD1hGP84aZCthZWM21gHMtySnk/q9Ce5K8bwaYDVRRU1vHAJfbS5PVjk/mfT3J4ePFmq8TTh8LKOj7fWkxD01D8fbwd+jcRR3zN62zp6elG5+N3Xcu2l3DHwgyeuyWdGedYkz6TuqPNvLY2j2dW7OZg7VHiw/z5eO5k/Hy8SP/Tl9x8fi9+f/lgh2z7mNLqeqbN+5qBPUJ5Y875J9TQa+obSf/vL7lseAIfbSrk9ol9+M2sQQD868ud/OvLXAAm9ovmd7MGMyghFBHhyNEm5q/cw7Nf72FCajTP35p+Qm+1tLqeWU+uIjLIjw9+MZEgv9b7cvvKa7n48ZUMSQzj1TvGndD7brYZPsgq4IlluVTVNTJ1QBxTBsRSVtPAih1lrNt7iIggX3pHB+PjLazde4hmm2FoYhgPXjKQC/qfen5i+ryvSYwIZOHtY1m9+yA3vbCW689L5s9XDeuUv3V7bdxfwXXz1zAmJZJnbhrN+L8s4+rRSfyPFUdtQxOvr91P2eEGauqbeG9jPr2jg1l013gig/04crSJt9Yf4Lw+UQzpGQ5AWU0DF/xtOdMHx/PU7FGAvbQ1bd4KQv19WfLLSTQ02Vjw7V7+/vkOJqfFEB3sx+dbS8h4ZDrB/j4cPNzA+L8so7HZ8KsZ/Zk7LY3lO0r5yYvreeHWdKYN6pz/JyKSaYxJP7ndYT1+EQkAVgL+1nbeMcb8l4i8BFwIHBuWcJsxJstRcSjnqm9s5qXv9hEd7MeUAe07gXkuAv28uXNyX24Yl8LiDQWk944kJsTeM75wQCwfby7id7MGndO3i/b62+c7aGi08bcfjTjlxGlogC+XDOnB4g32On/Lk7k3jE1ha2E114xOPF5uOSbIz4f7pvfnzsl98fP2OqVEERcWwL+uG8XNC9by28Vb+Oe1I1s9afvEsly8vODZm8acUnLx9hKuHp3E1aOTMMacsI07J/c9pe3g4QY+3lLEc6v2cMuCdVzQP5ZHfzD4eO+3pLqeXaWHuTbdXrY7PzWam8f34uXV+7hxXC8G92z7m0JHnHzCuqGpmV+9tYm4UH/+78bRRAT5MXNID5ZsKuQ/Lx9MgK83j364lbcz8wnw9SLE34fRKZE8NXvU8W+mQX4+3DaxzwnbiQ315/ZJvXl6+W6GJ4Zz5+Q+vLl+PwcO1fHiT4bi5SUE+nnz86n9iA315+F3N2MzcOXInsf/9tEh/swalsCKHWXcOqE3ABNTYwgP9OXjzUWdlvjb4shSTwNwkTHmsIj4At+IyKfWc/9hjHnHgdtW3cB3u8t55L1s9pTX8tDMgfh6O34sQZCfDzeN73VC2+XDE1i6rYSMvArG9omitqGJ4up6+sYEd1rdf3N+Je9k5vPTC/vSJya41ddcPTqRDzcVEuDrRXrv70sNcWEBPHfLKZ2yE4Scpj4+KS2G+6f3Z97SnRRW1fPE9SNJCP++Nr+rtIb3swqYM7kvcWEBp91Oa3+Pk9uiQ/y55fze9jLb6jyeXJbL3a9m8vl9FyAix4enTkiNOf6e+6f354OsAv7w0VbemDO+Q3/3rAOVxJ50/iEz7xA/eXE9f//xCC4Z0gOA51ftZW95LS/fPvZ4Ir9mTBLvZxXy5fYSQvx9eDszn59NSeXBmQPPKoZfTE1jd2ktf/5kO5vyK1m79xBj+0Qx5aRvP9emJxMW4Mt/fZjNzeef+O/yf64aRnV9I+GB9vNbfj5eXDIknk+2FFPf2EyAr+PKPQ77n2jsjl2i5mvdun9dSXWKhd/t44bn1tJkM7x8+1jumZLqtFimD4onwNeLJZsL+SqnhGn//Jpp//yaSX9dzp+WbOvwtBDGGP740TZiQvz4xdR+bb5uUr8Y4sP8mZAa0+n/qedOS2PetSPILqji0idW8XmL0SuPf5lLkK83P72wc4+Bv4/9W9bvLx/MzpLDfLvLfm3Gt7sOEhnky+CE73v24UG+/PriAazde4hPtpx+ZI3tNBe0bS+q5tp/r+buVzJPGI204Nt9VNc3cf+bWeQUV1NYWcf/frWLS4bEn1CKmpAaQ4+wAF5ZncdvF2+hX1wIc6elnfW+B/p588xNo3lw5gA+3lJEWU0DD80c0OoH2syhPVj72+mM6RV1Qnuwv88JH9AAlw3vyeGGJlbsKDvrmM6GQ7tgIuItIllAKbDUGLPWeurPIrJZRB4XkVYHWIvIHBHJEJGMsjLH/hFU51u8IZ9hieF8ft8FrdaAu1Kwvw8XDYxj0br93P5SBqEBPvzn5YMZ0COUV1bnccXT37Jy57n/G/tocxEZeRU8cPGA46OTWuPj7cVbPz2fx65xTJ376tFJfDx3MkmRgfz0lUzmLtrIt7vK+XhzET+Z2MdhJ1Z/MKIn0cF+vPjtXowxfLernPNTo08pq80em8KghDAeeHsTv1m8pdWL0BZvyGfkH79gS/6pzzU0NXP/m1k02WxsKagiM68CsJeevthazA9H9CTE34e7Xs7gkfezsRnDI5edeF7H20u4anQia/ceori6nr//aPg5fwiLCD+b0o/X7hzHn68aekpiPxcTUqNJjAjk75/nUN/ouHmcHJr4jTHNxpiRQBIwVkSGAr8BBgLnAVHAQ228d74xJt0Ykx4b69zEoc5OQ1Mz24qqmdgvhkA/x45OaK/rz0vBS4S509JYMncSt0/qw4LbzuObh6bSKzqYOxau56NNhWf9exuamnnsk+0MTghr15DKXtHBxIWevtzSEX1igll8z0Tun96fT7OLuPH5tYQG+HDX5L4O22aArzc3ju/FVztKWbGzjMKqes5vUeY5xttLmH/zGC4fnsB7G/O5/KlvuPmFtRw4ZB8T/1l2EQ+8vYnq+ibes655aGneFzvJKa7hqdmjCQvwYYF1DcO7G/JpbDb88qJ+PHvzGEqqG/gqp5SfT+1HclTQKb/nR2OS8BL7+YvOGOk1ITWGG8f1OvML28HX24v/uXoYu8tqeWJZbqf8ztZ0yQVcxphKYDkw0xhTZJWBGoAXgbFdEYPqOtuLamhsNoxMjnB2KMdd0D+W7X+cya9m9D9hqFxcWABvzBnPqJRI5r6x8fhFNe31VkY+hVX1/GbWwG5zJayfjxf3Tk9jyS8nMzktht9cOojwoLa/iXSGm8an4OMlPPTOZgAmpka3+rrkqCD+/uMRrP3tdH43axAb91dyyb9W8qcl2/jloo2MTI5gYr9oPt9afEIpZ93eQ8xftYfZY1O4bHgCs8em8Fl2MfkVR3hj3QHSe0WSFh/KqJRI/nXdSGYN68GcC1r/sEuNDWHZr6fw0FnW9bvKhf1juS49mWe/3s2mA5UO2YbDEr+IxIpIhHU/EJgB5IhIgtUmwJVAtqNiUM6Rtd/+Fbw7JX6gzRE94YG+vHz7WAbEh/LEl7ntvpL1aJONf6/YzeiUCCb1O7WH62wDeoTyyh3juGFcisO3FRcawOXDe1Ja00BCeECbJ7iPCQ/05a4L+vL5/RdwXu8oXvhmL2lxobz4k7FcMSKRgso6thbaJzIzxvCnJdtIjAjkkcvsQ2BvmdAbEeH+N7PYU17L7LHf7+OsYQn8341jTlvC6RMT3G0+qFvzu8sHERcawH+8s4mGps4v+Tiyx58ALBeRzcB67DX+JcBrIrIF2ALEAP/twBiUE2zKryI+zJ8e4Y4raXS2AF9vbp/Yhx0lNazbe6hd71m8IZ+Cyjp+OS2tS64K7u5+MrE3YC99tPfvkRgRyEs/OY/X7xzHorvGEx7oy7RBcXgJfLGtBIC1ew+xpaCKe6akHh8OmRgRyMwhPVi/r4LQAB9mDUtwyD45S1iAL3+5ehg7Sw6zbHtpp/9+R47q2WyMGWWMGW6MGWqM+aPVfpExZpjVdlOLkT/KTWw6UMmIpAhnh3HWfjCiJ2EBPryyJq/V56uONJJdUIUxhsZmG0+v2MXwpPBThvB5quFJEfz3lUPPegSXiDChX8zxclR0iD/pvaOOz6vz/Ko9RAX7cc3oE6fzOPZBc9WoxG5zLqkzTR0Yx8dzJznkQ02nbFCdqupII3vKa1udc6e7C/Tz5sfpySz8bh+lNfWnnIR94J1NLN1WQr+4EIYlhnPgUB3/efkQ7e23cPI1FOfqkiE9+NOSbSzPKeXL7aXcOy3tlNLNmF6R/N+No5nQxvkEd3DsauHOprNzqk61uaAS6H71/fa6cVwKTTbDm+sOnNC+q7SGpdtKmDE4ntAA+7S6gxPCmD7I8RNqeaKLrak97nszCz8fr1MufgL7N4VZwxKICOr6+X9cnfb4VafK2l+JCAxLckxPxdH6xoYwOS2G19ft554pqfhYVxvPX7mHAF8vHrt6GNEh/uwqPUx4oK/29h0kOSqIwQlhbCuqZvbY5OPTb6jOoT1+1ak25VeSGhtC2GkuZOrubh7fi6Kqep5daZ/Hv6S6nvc3FvLjMclEWwmoX1xIly/u4mlmDeuBl8Adkxx3DYKn0h6/6jTGGLIOVHGhi5/snD4onh+M6MnfP9+Bv48X5YeP0mSzcefkPmd+s+o0d13Ql4uH9KBfXMiZX6zOiiZ+1WkKq+opP9zAyGTXLPMc4+UlPH7tCJqabfz3x9vx8/Hi0mEJ9Io+/dh01bn8fbzPuMaAOjda6lGdJmOfffz7CBc9sduSj7cXT84exYzB8TQ22/hpG1eBKuWKtMevOsX+g0f440fb6BUddNqVmVyJr7cXz9w4msLKelKiT53zRSlXpT1+1WEVtUe57cV1NBvDgtvOw8/Hff5Z+Xh7adJXbkd7/KpDjjbZuPPlDPIr63j9znGkxuqJOKW6O/fpmimnWJVbRmZeBX+5ahjpvTs+H7lSyvE08asOycyrwMdLuGy4e02SpZQ708SvOiQzr4IhieEOXR9UKdW5NPGrc9bYbGNTfiWjUyKcHYpS6ixo4lfnLKeohvpGG2N6dXz5OqVU19HEr85ZZp79gi1N/Eq5Fk386pxl7q+kZ3gACeGBzg5FKXUWNPGrc7Yhr4JR2ttXyuU4crH1ABFZJyKbRGSriPzBau8jImtFZJeIvCkiuoqCCyquqqegso4xKZr4lXI1juzxNwAXGWNGACOBmSIyHvgr8Lgxph9QAdzhwBiUg2zYXwFofV8pV+TIxdZNi4XUfa2bAS4C3rHaFwJXOioG5TiZeRX4+3gxKME9JmRTypM4tMYvIt4ikgWUAkuB3UClMabJekk+kNjGe+eISIaIZJSVlTkyTHUOMvMqGJEU4VYTsinlKRz6v9YY02yMGQkkAWOBgWfx3vnGmHRjTHpsrGuv6ORuquoa2VpYxWgt8yjlkrqku2aMqQSWA+cDESJybFbQJKCgK2JQnefdzHwamw2X6/w8SrkkR47qiRWRCOt+IDAD2I79A+BH1stuBT5wVAyq89lshlfX5DE6JYKhia69xKJSnsqRPf4EYLmIbAbWA0uNMUuAh4BficguIBp4wYExqE723e6D7Cmv5ebzezk7FKXUOXLYQizGmM3AqFba92Cv9ysX9MqafUQF+zFrmJZ5lHJVOiRDtVthZR1Lt5Vw3XnJ+PvoNMxKuSpN/KrdFq3bjwFuGJvi7FCUUh2giV+1yze55Ty7cg8zBsWTHKWLjyvlyjTxqzNas+cgd768nr4xwfz1muHODkcp1UGa+NVpbTpQye0vrSc5MohX7xxHZLDOqaeUq9PEr07ruVV7CPT15rU7xxET4u/scJRSnUATvzqtnSU1jEqJIC4swNmhKKU6iSZ+1abGZht7y2vpFxfq7FCUUp1IE79q077yWhqbDf3jQ5wdilKqE2niV23KLbUvp9A/Xnv8SrkTTfyqTTtLahCB1Fjt8SvlTjTxqzbllhwmJSqIQD+dnkEpd6KJX7VpZ0kNaXpiVym3o4lftepok31ET5qe2FXK7WjiV63ad7CWJpuO6FHKHWniV63KLbGP6NFSj1LuRxO/atXOkhq8BPrFaY9fKXejiV+1Kre0hpSoIAJ8dUSPUu7GkYutJ4vIchHZJiJbReReq/1RESkQkSzrNstRMahzt7PkMGl64ZZSbslha+4CTcCvjTEbRCQUyBSRpdZzjxtj/uHAbasOONpkY195LRcPjnd2KEopB3DkYutFQJF1v0ZEtgOJjtqe6jhjDCLSYkSP9viVckeO7PEfJyK9gVHAWmAi8AsRuQXIwP6toKIr4lBtK66qZ/q8r7EZc7yur2P4lXJPDj+5KyIhwLvAfcaYauAZIBUYif0bwT/beN8cEckQkYyysjJHh+nxcoqrOdzQxPRB8UwdEMeN41IYoD1+pdySQ3v8IuKLPem/ZoxZDGCMKWnx/HPAktbea4yZD8wHSE9PN46MU0FRVT0AD106kMSIQCdHo5RyJEeO6hHgBWC7MWZei/aEFi+7Csh2VAyq/Yoq6/ASiAvV5RWVcneO7PFPBG4GtohIltX2W2C2iIwEDLAP+KkDY1DtVFhVT2yoP77eemmHUu7OkaN6vgGklac+cdQ21bkrrqonIVxLPEp5Au3eKQAKq+roGaELqivlCTTxK4wxFFXW0yNMe/xKeQJN/IrquibqGpu1x6+Uh9DEryisqgPQGr9SHkITv6LoWOLXHr9SHkETv6Kw0n7xVkK4Jn6lPIEmfkVxVT3eXkJcqCZ+pTyBJn5FYVUd8aH+eHu1dtmFUsrdaOJX9qGcWuZRymNo4lcUVdWRoBOzKeUxNPF7OGMMRVX19NQev1IeQxO/h6s40khDk03H8CvlQTTxe7jCymMXb2mPXylPoYnfwxVbC7BojV8pz6GJ38Mdu2pXa/xKeQ5N/B6usKoeHy8hOkRX3lLKU2ji93BFlXXEhwXoxVtKeZB2J34RCRSRAY4MRnW9oqp6nY5ZKQ/TrsQvIj8AsoDPrMcjReRDB8alukhRVT09dCinUh6lvT3+R4GxQCWAMSYL6HO6N4hIsogsF5FtIrJVRO612qNEZKmI5Fo/I885etUhjc02irXHr5THaW/ibzTGVJ3UZs7wnibg18aYwcB44OciMhh4GFhmjEkDllmPlRNsK6zmaLONYYnhzg5FKdWF2pv4t4rIDYC3iKSJyFPAd6d7gzGmyBizwbpfA2wHEoErgIXWyxYCV55L4KrjMvIqAEjvFeXkSJRSXam9if+XwBCgAXgdqALua+9GRKQ3MApYC8QbY4qsp4qB+DbeM0dEMkQko6ysrL2bUmchM+8QiRGBOjOnUh7G50wvEBFv4GNjzFTgd2e7AREJAd4F7jPGVIt8P2zQGGNEpNWSkTFmPjAfID09/UxlJXWWjDGs31fBxNRoZ4eilOpiZ+zxG2OaAZuInHUhWER8sSf914wxi63mEhFJsJ5PAErP9veqjjtwqI6ymgbG9NYyj1Ke5ow9fsthYIuILAVqjzUaY+a29Qaxd+1fALYbY+a1eOpD4FbgMevnB2cbtOq4jLxDAKT30kFVSnma9ib+xdbtbEwEbsb+gZFltf0We8J/S0TuAPKAa8/y96pOkJFXQai/D/3jQ50dilKqi7Ur8RtjFoqIH9DfatphjGk8w3u+AdqaB2Ba+0NUjpC5r4LRvSJ1qgalPFB7r9ydAuQCTwP/B+wUkQscF5ZypKojjewoqdEyj1Ieqr2lnn8CFxtjdgCISH9gETDGUYEpx9mw3z5+f0xvTfxKeaL2juP3PZb0AYwxOwFfx4SkHC0j7xDeXsLI5Ahnh6KUcoL29vgzROR54FXr8Y1AhmNCUo6WmVfBkJ5hBPm19/ArpdxJe3v89wDbgLnWbZvVplyMMYZthdU6P49SHqy9XT4f4Ilj4/Gtq3l1ySYXVFRVT3V9EwMTwpwdilLKSdrb418GtJy0PRD4svPDUY6WU1wNwKAeOn5fKU/V3sQfYIw5fOyBdT/IMSEpR9peVAPAAE38Snms9ib+WhEZfeyBiKQDdY4JSTnS9qJqkqMCCQ3QQVlKear21vjvA94WkULrcQJwnUMiUg6VU1zDwB5a31fKk522xy8i54lID2PMemAg8CbQiH3t3b1dEJ/qRPWNzewpO6z1faU83JlKPc8CR63752OfZO1poAJrrnzlOnJLDmMz6IgepTzcmUo93saYQ9b964D5xph3gXdbzLipXMT2YyN6NPEr5dHO1OP3FpFjHw7TgK9aPKeXfbqY7UXVBPp6kxKlA7KU8mRnSt6LgK9FpBz7KJ5VACLSD/u6u8qF5BTV0L9HqE7FrJSHO23iN8b8WUSWYR/F84Ux5tjat17YF2BXLsIYQ05xNZcM6eHsUJRSTnbGco0xZk0rbTsdE45ylJLqBiqONGp9XynV7gu4lIs7dmJ3oA7lVMrjOSzxi8gCESkVkewWbY+KSIGIZFm3WY7avjpRjjVVg168pZRyZI//JWBmK+2PG2NGWrdPHLh91cLWwioSIwIJD9KpGpTydA5L/MaYlcChM75QdYnsgiqGJmpvXynlnBr/L0Rks1UKanPRVxGZIyIZIpJRVlbWlfG5naq6RvYdPMLwpAhnh6KU6ga6OvE/A6QCI4Ei7Iu4t8oYM98Yk26MSY+Nje2i8NzT1gL7JRdDddUtpRRdnPiNMSXGmGZjjA14Dhjbldv3VFusxK/LLSqloIsTv4gktHh4FZDd1mtV59lSYD+xGxXs5+xQlFLdgMPm2xGRRcAUIEZE8oH/AqaIyEjAAPuAnzpq++p7emJXKdWSwxK/MWZ2K80vOGp7qnXHTuz+OD3Z2aEopboJvXLXzemJXaXUyTTxuzk9sauUOpkmfjenJ3aVUifTxO/mthRUaW9fKXUCTfxurKqukbyDRxiWpIlfKfU9TfxuTE/sKqVao4nfjb2fVUCArxcjkyOcHYpSqhvRxO+myg838H5WIdeMTiI8UKdiVkp9TxO/m3p1TR5Hm2zcPqmPs0NRSnUzmvjdUH1jM6+uyWPqgFhSY0OcHY5SqpvRxO+GPtxUSPnho9wxqa+zQ1FKdUOa+N2MMYYF3+xlYI9QJvaLdnY4SqluSBO/m9lSUEVOcQ23TuiNiDg7HKVUN6SJ382s3GlfpnLG4HgnR6KU6q408buZlbnlDOkZRkyIv7NDUUp1U5r43cjhhiY25FUwOU3XKFZKtU0TvxtZs/sgTTbDBf1jnB2KUqob08TvRlbmlhHo682YXpHODkUp1Y1p4ncjq3LLGd83Cn8fb2eHopTqxhyW+EVkgYiUikh2i7YoEVkqIrnWT+2adpIDh46wt7xW6/tKqTNyZI//JWDmSW0PA8uMMWnAMuux6gSrcssBuKC/Jn6l1Ok5LPEbY1YCh05qvgJYaN1fCFzpqO17mpU7y+gZHkBqbLCzQ1FKdXNdXeOPN8YUWfeLgTavMhKROSKSISIZZWVlXROdi2pstvHt7nImp8Xq1bpKqTNy2sldY4wBzGmen2+MSTfGpMfGavnidDL2VVBT38TUgXHODkUp5QK6OvGXiEgCgPWztIu375aWbS/Bz9uLyWk6fl8pdWZdnfg/BG617t8KfNDF23dLX+WUMj41mmB/H2eHopRyAY4czrkIWA0MEJF8EbkDeAyYISK5wHTrseqAPWWH2VNey/RBWuZRSrWPw7qIxpjZbTw1zVHb9ETLtturZRdpfV8p1U565a6LW5ZTwsAeoSRFBjk7FKWUi9DE78KqjjSyfl+F9vaVUmdFE78L+zq3jGabYdogXXRFKdV+mvhdVFOzjbczDhAV7MfI5Ahnh6OUciGa+F1QfWMzd7+ayarccn42JRVvL71aVynVfjrw28VU1TVy58L1ZORV8Kcrh3Lz+F7ODkkp5WI08buYZ7/ezYb9lTw1exSXD+/p7HCUUi5ISz0uZktBFQN7hGrSV0qdM038LmZHcQ0DeoQ6OwyllAvTxO9CDtUepbSmgUE9wpwdilLKhWnidyE5xdUA2uNXSnWIJn4XsqO4BoCBmviVUh2gid+F7CiuISrYj9hQf2eHopRyYZr4Xcj24hoGxIfq8opKqQ7RxO8ibDZDbomO6FFKdZwmfhdxoOIIR442MyhBE79SqmM08buIHOvE7gAdyqmU6iBN/C4ip6gGEegfH+LsUJRSLs4pc/WIyD6gBmgGmowx6c6Iw5XsKKmmV1QQQX46vZJSqmOcmUWmGmPKnbh9l5KjUzUopTqJlnpcQH1jM/vKa7W+r5TqFM5K/Ab4QkQyRWROay8QkTkikiEiGWVlZV0cXveSW3IYm4FB2uNXSnUCZyX+ScaY0cClwM9F5IKTX2CMmW+MSTfGpMfGxnZ9hN3Iih2lAAxLCndyJEopd+CUxG+MKbB+lgLvAWOdEYcraGq28fq6/UxOiyEpMsjZ4Sil3ECXJ34RCRaR0GP3gYuB7K6Ow1V8ub2Uoqp6XWJRKdVpnDGqJx54z5pvxgd43RjzmRPicAmvrsmjZ3gAFw2Mc3YoSik30eWJ3xizBxjR1dt1RbvLDvPNrnL+45IB+HjrACylVOfQbNKNvbI6D19v4dr0ZGeHopRyI5r4uwFjDMaYE9oOHDrCu5n5zBqWoPPvK6U6lSZ+J7PZDHe9nMnUf6xg5U779QrZBVVc/cx3iMDPpvRzcoRKKXejE7842YJv9/Ll9hJiQvy5ZcE6ZgyOZ/Xug4QF+PDaPRPoH68XbSmlOpf2+J0ou6CKv36Ww8WD4/nmoancNz2Nr3eUkRQZyOKfTdSkr5RyCO3xO8mRo03MXbSR6GB//nrNcAJ8vblven9uGJdCWIAvAb7ezg5RKeWmNPE7Qd3RZu5+dQN7D9by2p3jiAz2O/5cXGiAEyNTSnkCTfxdrLq+kTteWk9GXgWPXT2MCakxzg5JKeVhNPF3oaojjdzw/Bp2ltTw1OxRXD68p7NDUkp5IE38XcQYw0PvbmZHcQ3P3ZrO1AE6BYNSyjl0VE8XeXP9AT7bWsx/XDJAk75Syqm0x+8Axhi2F9Xg7SWkxYWwp7yWP3y0jYn9orlrcl9nh6eU8nCa+DtR3sFa3so4wJLNReQdPAJAqL8P/r5eBPh6Me/akXh5iZOjVEp5Ok38naCspoGnvsrl9bX7McCE1GjuuTAVX28vNh6oYFthNfdO7098mA7VVEo5n8ck/l2lNXyxrYTVuw+Stb+S81OjefSHQ+gZEQjAvvJadpTUMGVALP4+rV88ZbMZ9pTXsrWwiu1FNRRX1VFcXc+W/Crqm2zMHpvM3IvSiGuR4K8Zk9Ql+6eUUu3l9ol//8EjzFu6gw82FWIMDIgPZcaQeD7dUsz0eV9z+8Q+bMqvZFVuOQCJEYHMndaPy4b3JL/iCPvKa8kuqGbjgQo2HajicEMTAH7eXsSH+9MjLIDLh/dkzoV9SY0NceauKqVUu8jJ0wF3R+np6SYjI+Os3/fUslyeWJaLj7dw24Q+3D6p9/ErYw8cOsLvP8hmxY4yEsIDmD02hYE9Qnl6xW42Hag84fd4ewmDEkIZmRzB8KQIhiWGkxYXooujKKW6NRHJNMakn9zu1j3+pKhArjsvmbnT0k6prydHBfHibedx4FAdPSMCjifxGYPj+SqnlO1F1aREB9MnOpjUuGCC/Nz6T6WU8iBO6fGLyEzgCcAbeN4Y89jpXn+uPX6llPJkbfX4u7xWISLewNPApcBgYLaIDO7qOJRSylM5o0g9FthljNljjDkKvAFc4YQ4lFLKIzkj8ScCB1o8zrfaTiAic0QkQ0QyysrKuiw4pZRyd912WIoxZr4xJt0Ykx4bG+vscJRSym04I/EXAMktHidZbUoppbqAMxL/eiBNRPqIiB9wPfChE+JQSimP1OWD040xTSLyC+Bz7MM5FxhjtnZ1HEop5amcclWSMeYT4BNnbFsppTydS0zZICJlQN5ZvCUGKHdQON2B7p9r0/1zba60f72MMaeMjnGJxH+2RCSjtavV3IXun2vT/XNt7rB/3XY4p1JKKcfQxK+UUh7GXRP/fGcH4GC6f65N98+1ufz+uWWNXymlVNvctcevlFKqDZr4lVLKw7hd4heRmSKyQ0R2icjDzo6no0QkWUSWi8g2EdkqIvda7VEislREcq2fkc6O9VyJiLeIbBSRJdbjPiKy1jqGb1pTe7gkEYkQkXdEJEdEtovI+W527O63/l1mi8giEQlw5eMnIgtEpFREslu0tXq8xO5Jaz83i8ho50V+dtwq8bvpIi9NwK+NMYOB8cDPrX16GFhmjEkDllmPXdW9wPYWj/8KPG6M6QdUAHc4JarO8QTwmTFmIDAC+366xbETkURgLpBujBmKfQqW63Ht4/cSMPOktraO16VAmnWbAzzTRTF2mFslftxwkRdjTJExZoN1vwZ74kjEvl8LrZctBK50SoAdJCJJwGXA89ZjAS4C3rFe4sr7Fg5cALwAYIw5aoypxE2OncUHCBQRHyAIKMKFj58xZiVw6KTmto7XFcDLxm4NECEiCV0SaAe5W+Jv1yIvrkpEegOjgLVAvDGmyHqqGIh3Vlwd9C/gQcBmPY4GKo0xTdZjVz6GfYAy4EWrlPW8iATjJsfOGFMA/APYjz3hVwGZuM/xO6at4+Wy+cbdEr/bEpEQ4F3gPmNMdcvnjH1MrsuNyxWRy4FSY0yms2NxEB9gNPCMMWYUUMtJZR1XPXYAVq37CuwfcD2BYE4tk7gVVz5eLblb4nfLRV5ExBd70n/NGLPYai459rXS+lnqrPg6YCLwQxHZh70sdxH2mniEVToA1z6G+UC+MWat9fgd7B8E7nDsAKYDe40xZcaYRmAx9mPqLsfvmLaOl8vmG3dL/G63yItV834B2G6MmdfiqQ+BW637twIfdHVsHWWM+Y0xJskY0xv7sfrKGHMjsBz4kfUyl9w3AGNMMXBARAZYTdOAbbjBsbPsB8aLSJD17/TY/rnF8WuhreP1IXCLNbpnPFDVoiTUvRlj3OoGzAJ2AruB3zk7nk7Yn0nYv1puBrKs2yzstfBlQC7wJRDl7Fg7uJ9TgCXW/b7AOmAX8Dbg7+z4OrBfI4EM6/i9D0S607ED/gDkANnAK4C/Kx8/YBH28xWN2L+x3dHW8QIE+yjC3cAW7KObnL4P7bnplA1KKeVh3K3Uo5RS6gw08SullIfRxK+UUh5GE79SSnkYTfxKKeVhNPErtyYizSKS1eJ22gnRRORuEbmlE7a7T0RizuF9l4jIH6wZIT/taBxKtcbnzC9RyqXVGWNGtvfFxph/OzCW9piM/QKoycA3To5FuSnt8SuPZPXI/yYiW0RknYj0s9ofFZEHrPtzrXUQNovIG1ZblIi8b7WtEZHhVnu0iHxhzU3/PPaLe45t6yZrG1ki8qw1ffjJ8VwnIlnYpzn+F/Ac8BMRcekrz1X3pIlfubvAk0o917V4rsoYMwz4X+zJ9mQPA6OMMcOBu622PwAbrbbfAi9b7f8FfGOMGQK8B6QAiMgg4DpgovXNoxm48eQNGWPexD7zarYV0xZr2z88911XqnVa6lHu7nSlnkUtfj7eyvObgddE5H3s0y2AfQqNawCMMV9ZPf0w7PPuX221fywiFdbrpwFjgPX26WwIpO1J2foDe6z7wca+/oJSnU4Tv/Jkpo37x1yGPaH/APidiAw7h20IsNAY85vTvkgkA4gBfERkG5BglX5+aYxZdQ7bVapNWupRnuy6Fj9Xt3xCRLyAZGPMcuAhIBwIAVZhlWpEZApQbuzrI6wEbrDaL8U+GRvYJ/f6kYjEWc9FiUivkwMxxqQDH2Of3/5v2CcYHKlJXzmC9viVuwu0es7HfGaMOTakM1JENgMNwOyT3ucNvGotnyjAk8aYShF5FFhgve8I30/X+wdgkYhsBb7DPmUxxphtIvII8IX1YdII/BzIayXW0dhP7v4MmNfK80p1Cp2dU3kka/GXdGNMubNjUaqraalHKaU8jPb4lVLKw2iPXymlPIwmfqWU8jCa+JVSysNo4ldKKQ+jiV8ppTzM/wMGDIJ3F16UEwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ddpq_agent import Agent\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "agent = Agent(device, \n",
    "              n_agents, \n",
    "              state_size, \n",
    "              action_size,\n",
    "              random_seed=47,\n",
    "              buffer_size=int(1e5), # replay buffer size\n",
    "              batch_size=128,       # minibatch size\n",
    "              gamma=0.98,           # discount factor\n",
    "              tau=1e-3,             # for soft update of target parameters\n",
    "              fc1_units_actor=512,  # output dim of first FC layer of actor\n", 
    "              fc2_units_actor=64,   # output dim of second FC layer of actor\n",
    "              fcs1_units_critic=512,# output dim of first FC layer of critic\n",
    "              fc2_units_critic=64,  # output dim of second FC layer of critic\n",
    "              lr_actor=5e-4,        # learning rate of the actor \n",
    "              lr_critic=5e-4,       # learning rate of the critic\n",
    "              weight_decay=0)       # L2 weight decay\n",
    "\n",
    "\n",
    "# train the agent\n",
    "ddpg_train(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch the Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode Score (averaged across agents): 38.973499128874394\n"
     ]
    }
   ],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
    "\n",
    "env_info = env.reset(train_mode=False)[brain_name]\n",
    "states = env_info.vector_observations\n",
    "episode_scores = np.zeros(n_agents) # score for each agent within episode\n",
    "agent.reset()\n",
    "\n",
    "for t in range(1000):\n",
    "    actions = agent.act(states, add_noise=False)\n",
    "    env_info = env.step(actions)[brain_name]\n",
    "    next_states = env_info.vector_observations\n",
    "    rewards = env_info.rewards\n",
    "    dones = env_info.local_done\n",
    "    \n",
    "    # Sum up rewards separately for each agent\n",
    "    episode_scores += np.array(rewards)\n",
    "\n",
    "    # Prepare for next timestep of iteraction\n",
    "    states = next_states  # new states become the current states\n",
    "\n",
    "    # Check if any of the agents has finished. Finish to keep all\n",
    "    # trajectories in this batch the same size.\n",
    "    if np.any(dones):\n",
    "        break\n",
    "\n",
    "episode_score = np.mean(episode_scores) # Summary of scores for this episode\n",
    "print(\"Episode Score (averaged across agents): {}\".format(episode_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}