ipython_laser_learn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division, print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*original post: *[Simple Reinforcement Learning with Tensorflow](https://medium.com/@awjuliani/simple-reinforcement-learning-with-tensorflow-part-4-deep-q-networks-and-beyond-8438a3e2b8df)\n",
    "## Double Dueling Deep Q Network Model\n",
    "Model with laser vector input and output action scores\n",
    "\n",
    "**Separated network**\n",
    "* target network for estimate the target q value in training (the label)\n",
    "* main network for estimate the predict q value in training (the output)\n",
    "\n",
    "Why not use just use one network for both estimations? The issue is that at every step of training, the Q-network’s values shift, and if we are using a constantly shifting set of values to adjust our network values, then the value estimations can easily spiral out of control. The network can become destabilized by falling into feedback loops between the target and estimated Q-values. In order to mitigate that risk, the target network’s weights are fixed, and only periodically or slowly updated to the primary Q-networks values. In this way training can proceed in a more stable manner.\n",
    "\n",
    "**Double Q**\n",
    "$$ Q-target = r + \\gamma Q(s', \\operatorname{arg\\,max}_a Q(s', a, \\theta), \\theta ') $$\n",
    "\n",
    "* $\\theta$ is the parameters of the main model\n",
    "* $\\theta'$ is the parameters of the target model\n",
    "\n",
    "**Dueling Q**\n",
    "$$ Q(s, a) = V(s) + A(a) $$\n",
    "\n",
    "* $V(s)$ tells how good the state is\n",
    "* $A(a)$ tells how better taking a certain action would be compared to others\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "import keras\n",
    "from keras import optimizers\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Lambda\n",
    "import keras.backend as K\n",
    "\n",
    "class Qnet:\n",
    "    \n",
    "    def __init__(self, numstate, num_actions):\n",
    "        self.input_size = numstate\n",
    "        self.output_size = num_actions\n",
    "        self.model = self.create_model()\n",
    "        self.target_model = self.create_model()\n",
    "    \n",
    "    def create_model(self):\n",
    "\n",
    "        inp = Input(shape=(self.input_size[0], ))\n",
    "        inner = Dense(300, activation='relu')(inp)\n",
    "        inner = Dense(300, activation='relu')(inner)\n",
    "        out = Dense(self.output_size + 1)(inner)\n",
    "\n",
    "        # out = value + avantage - mean(advantage)\n",
    "        out = Lambda(lambda a: K.expand_dims(a[:, 0], axis=-1) + a[:, 1:] - K.mean(a[:, 1:], keepdims=True, axis=1))(out)\n",
    "\n",
    "        model = Model(inp, out)\n",
    "        \n",
    "        model.summary()\n",
    "\n",
    "        optimizer = optimizers.Adam(0.0001)\n",
    "        model.compile(loss=\"mse\", optimizer=optimizer)\n",
    "        return model\n",
    "    \n",
    "    def get_qvalues(self, states):\n",
    "        predicted = self.model.predict(states)\n",
    "        return predicted\n",
    "\n",
    "    def get_target_qvalues(self, states):\n",
    "        predicted = self.target_model.predict(states)\n",
    "        return predicted\n",
    "    \n",
    "    def get_actions(self, states):\n",
    "        qvalues = self.get_qvalues(states)\n",
    "        actions = np.argmax(qvalues, axis=1)\n",
    "        return actions\n",
    "    \n",
    "    def update_target_model(self, tau):\n",
    "        main_weights = self.model.get_weights()\n",
    "        target_weights = self.target_model.get_weights()\n",
    "        for i, layer_weights in enumerate(main_weights):\n",
    "            target_weights[i] *= (1-tau)\n",
    "            target_weights[i] += tau * layer_weights\n",
    "        self.target_model.set_weights(target_weights)\n",
    "        \n",
    "    def save(self, path):\n",
    "        self.target_model.save(path + \"/target.h5\")\n",
    "        self.model.save(path + \"/main.h5\")\n",
    "        \n",
    "    def load(self, path):\n",
    "        self.target_model.load_weights(path + \"/target.h5\")\n",
    "        self.model.load_weights(path + \"/main.h5\")\n",
    "        \n",
    "    def learn_on_minibatch(self, minibatch, gamma):\n",
    "        states = np.vstack(minibatch[:,0])\n",
    "        actions = minibatch[:, 1]\n",
    "        rewards = minibatch[:, 2]\n",
    "        dones = minibatch[:, 3]\n",
    "        newstates = np.vstack(minibatch[:, 4])\n",
    "\n",
    "        actions_newstate = self.get_actions(newstates)\n",
    "        target_qvalues_newstate = self.get_target_qvalues(newstates)\n",
    "        double_q = target_qvalues_newstate[range(target_qvalues_newstate.shape[0]), actions_newstate]\n",
    "\n",
    "        done_multiplier = 1 - dones\n",
    "        target_q = rewards + gamma * double_q * done_multiplier\n",
    "        \n",
    "        qvalues = self.get_qvalues(states)\n",
    "        for i in range(qvalues.shape[0]):\n",
    "            qvalues[i, actions[i]] = target_q[i]\n",
    "\n",
    "        loss = self.model.train_on_batch(states, qvalues)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qnet = Qnet([100], 11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience Replay\n",
    "To store the agent's experiences, and then randomly drawing batche of them to train the network. By keeping the network random, we can prevent the network from only learning about the immidiate actions and allow it to learn vastly from the past.\n",
    "\n",
    "Each of these experiences are stored as a tuple of `<state, action, reward, next_state>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "class ExperienceReplay:\n",
    "    \"\"\"\n",
    "    Class for storing experience\n",
    "    1 experience is an array of [state, action, reward, done, newstate]\n",
    "    \"\"\"\n",
    "    def __init__(self, output_dir, buffer_size):\n",
    "        self.buffer = []\n",
    "        self.buffer_size = buffer_size\n",
    "        self.output_dir = output_dir\n",
    "    \n",
    "    def add(self,experience):\n",
    "        if len(self.buffer) + len(experience) >= self.buffer_size:\n",
    "            self.buffer[0:(len(experience)+len(self.buffer))-self.buffer_size] = []\n",
    "            \n",
    "        self.buffer.extend(experience)\n",
    "\n",
    "    def sample(self, size):\n",
    "        return np.reshape(np.array(random.sample(self.buffer, min(size, len(self.buffer)))), [size, 5])\n",
    "\n",
    "    def save(self):\n",
    "        np.save(self.output_dir + \"/experience\", self.buffer)\n",
    "\n",
    "    def load(self, path):\n",
    "        self.buffer = np.load(path + \"/experience.npy\").tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the game evironment\n",
    "The custom game enviroments:\n",
    "\n",
    "![env3](screenshots/env3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import gym_gazebo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make('GazeboTurtlebotMazeColor-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network\n",
    "Setting the hyper parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "gamma = 0.95\n",
    "start_epsilon = 1.0\n",
    "end_epsilon = 0.05\n",
    "annealing_steps = 100000\n",
    "num_training_step = 1000\n",
    "num_pretrain_step = 10000\n",
    "tau = 0.001\n",
    "target_update_freq = 4\n",
    "online_update_freq = 4\n",
    "save_model_freq = 50\n",
    "buffer_size = 100000\n",
    "output_dir = \"output/test\"\n",
    "num_episode = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = start_epsilon\n",
    "epsilon_decay = (start_epsilon - end_epsilon)/annealing_steps\n",
    "\n",
    "episode = 0\n",
    "total_step = 0\n",
    "\n",
    "replay = ExperienceReplay(output_dir, buffer_size)\n",
    "\n",
    "while episode < num_episode:\n",
    "    state = env.reset()\n",
    "    replay_ep = ExperienceReplay(output_dir, buffer_size)\n",
    "    total_reward = 0\n",
    "    num_random_step = 0\n",
    "    total_loss = 0\n",
    "    num_training = 0\n",
    "    start_step = total_step\n",
    "    \n",
    "    if(total_step >= num_pretrain_step):\n",
    "        episode += 1\n",
    "        \n",
    "    for i in range(num_training_step):\n",
    "        \n",
    "        # get action\n",
    "        if(total_step < num_pretrain_step or np.random.rand(1) < epsilon):\n",
    "            action = np.random.randint(env.num_action)\n",
    "            num_random_step += 1\n",
    "\n",
    "        else:\n",
    "            action = qnet.get_actions(state.reshape(1, -1))[0]\n",
    "            \n",
    "        # get after take action\n",
    "        newstate, reward, done, _ = env.step(action)\n",
    "        if(newstate == []):\n",
    "            print(\"Terminate\")\n",
    "            # state = env.reset()\n",
    "            break\n",
    "        replay_ep.add(np.reshape(np.array([state, action, reward, done, newstate]), [1, 5]))\n",
    "        \n",
    "        # train\n",
    "        if total_step > num_pretrain_step:\n",
    "            if epsilon > end_epsilon:\n",
    "                epsilon -= epsilon_decay\n",
    "\n",
    "            if total_step % online_update_freq == 0:\n",
    "                train_batch = replay.sample(batch_size)\n",
    "                loss = qnet.learn_on_minibatch(train_batch, gamma)\n",
    "                total_loss += loss\n",
    "                num_training += 1\n",
    "                sys.stdout.write(\"\\rTrain step at {}th step | loss {} | epsilon {}\".format(total_step, loss, epsilon))\n",
    "                sys.stdout.flush()\n",
    "            \n",
    "            if total_step % target_update_freq == 0:\n",
    "\n",
    "                # print(\"Update target net\")\n",
    "                qnet.update_target_model(tau)\n",
    "                \n",
    "        total_step += 1\n",
    "        total_reward += reward\n",
    "        state = newstate\n",
    "        if done:\n",
    "            break\n",
    "            \n",
    "    replay.add(replay_ep.buffer)\n",
    "    \n",
    "    if(num_training == 0):\n",
    "        num_training = 1\n",
    "        \n",
    "    print(\"\\nDone epoch in {} steps, {} random steps, Total reward: {}, Total step: {}, Average loss: {}\".format(total_step - start_step, num_random_step, total_reward, total_step, total_loss/num_training))\n",
    "    \n",
    "    # save model\n",
    "    if(episode % save_model_freq == 0 and total_step > num_pretrain_step):\n",
    "        qnet.save(output_dir)\n",
    "        replay.save()\n",
    "        print(\"Save model at {}\".format(output_dir))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}