03_MAB_movielens_ucb1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03_MAB_movielens_ucb1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPj6eIzSNbFIvX/ttmJ+KDb",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pstanisl/mlprague-2021/blob/main/03_MAB_movielens_ucb1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VG26TaKfO8Dw"
      },
      "source": [
        "# Multi Armed Bandits - Upper Confidence Bounds\n",
        "\n",
        "Rather than performing exploration by simply selecting an arbitrary action, chosen with a probability that remains constant, the UCB algorithm changes its exploration-exploitation balance as it gathers more knowledge of the environment. It moves from being primarily focused on exploration, when actions that have been tried the least are preferred, to instead concentrate on exploitation, selecting the action with the highest estimated reward.\n",
        "\n",
        "#### Installation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DV6TkSLMLint"
      },
      "source": [
        "!rm -f ./utils.py\n",
        "!wget --no-check-certificate --no-cache --no-cookies \\\n",
        "    https://raw.githubusercontent.com/pstanisl/mlprague-2021/main/utils.py \\\n",
        "    -O ./utils.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsR9d0BTLl2h"
      },
      "source": [
        "##### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vGKUmVS0Ozou"
      },
      "source": [
        "from tqdm.notebook import trange\n",
        "from typing import Any, Dict, List, Text, Tuple\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import random\n",
        "import seaborn as sns\n",
        "import zipfile\n",
        "\n",
        "from utils import load_movielens_data\n",
        "from utils import plot_actions, plot_cumsum, plot_regret\n",
        "\n",
        "# Apply the default theme\n",
        "sns.set_theme()\n",
        "# Not show divide warnings\n",
        "np.seterr(divide='ignore', invalid='ignore')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DBtl9V0RPG9E"
      },
      "source": [
        "#### Downloading the [MovieLens](https://grouplens.org/datasets/movielens/) (100K) dataset.\n",
        "\n",
        "**Dataset info**\n",
        "\n",
        "MovieLens data sets were collected by the GroupLens Research Project\n",
        "at the University of Minnesota.\n",
        "\n",
        "This data set consists of:\n",
        "* 100,000 ratings (1-5) from 943 users on 1682 movies.\n",
        "* Each user has rated at least 20 movies.\n",
        "* Simple demographic info for the users (age, gender, occupation, zip)\n",
        "\n",
        "The data was collected through the MovieLens web site\n",
        "(movielens.umn.edu) during the seven-month period from September 19th,\n",
        "1997 through April 22nd, 1998. This data has been cleaned up - users\n",
        "who had less than 20 ratings or did not have complete demographic\n",
        "information were removed from this data set. Detailed descriptions of\n",
        "the data file can be found at the end of this file.\n",
        "\n",
        "Neither the University of Minnesota nor any of the researchers\n",
        "involved can guarantee the correctness of the data, its suitability\n",
        "for any particular purpose, or the validity of results based on the\n",
        "use of the data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2pZU_cijPEnO"
      },
      "source": [
        "print(\"Downloading movielens data...\")\n",
        "\n",
        "!wget --no-check-certificate \\\n",
        "    http://files.grouplens.org/datasets/movielens/ml-100k.zip \\\n",
        "    -O ./movielens.zip\n",
        "\n",
        "zip_ref = zipfile.ZipFile('movielens.zip', \"r\")\n",
        "zip_ref.extractall()\n",
        "\n",
        "print(\"Done. Dataset contains:\")\n",
        "print(zip_ref.read('ml-100k/u.info').decode())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wj_Kl6kmOrQH"
      },
      "source": [
        "#### Parameters -- Feel Free to Play Around"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "shTjKCBiOwWG"
      },
      "source": [
        "RANK_K = 20 # @param {type:\"integer\"}\n",
        "NUM_ACTIONS = 20 # @param {type:\"integer\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQc660eEPPLW"
      },
      "source": [
        "## Environment\n",
        "\n",
        "Implementation of the environment uses **MovieLens 100K dataset**. As described above, the dataset contains 100000 ratings from 943 users and 1682 movies. The environment can consider only the first $n$ of the dataset's movies. It can be set-up by `num_actions`. The number of \"known\" movies for the environment is equal to actions/arms.\n",
        "\n",
        "> Users without a rating (after selecting first $n$ movies) are removed from the environment.\n",
        "\n",
        "> In every step, the batch of users will be selected randomly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "encQfL8cPPri"
      },
      "source": [
        "class MovielensEnvironment(object):\n",
        "  def __init__(\n",
        "      self, \n",
        "      data_dir: Text,\n",
        "      rank_k: int, \n",
        "      batch_size: int = 1,\n",
        "      num_movies: int = 20\n",
        "  ):\n",
        "    \"\"\"Initializes the MovieLens Bandit environment.\n",
        "    Args:\n",
        "      data_dir: (string) Directory where the data lies (in text form).\n",
        "      rank_k : (int) Which rank to use in the matrix factorization.\n",
        "      batch_size: (int) Number of observations generated per call.\n",
        "      num_movies: (int) Only the first `num_movies` movies will be used by the\n",
        "        environment. The rest is cut out from the data.\n",
        "    \"\"\"\n",
        "    self._num_actions = num_movies\n",
        "    self._batch_size = batch_size\n",
        "    self._context_dim = rank_k\n",
        "\n",
        "    # Compute the matrix factorization.\n",
        "    self._data_matrix = load_movielens_data(data_dir)\n",
        "    # Keep only the first items.\n",
        "    self._data_matrix = self._data_matrix[:, :num_movies]\n",
        "    # Filter the users with no iterm rated.\n",
        "    nonzero_users = list(np.nonzero(np.sum(self._data_matrix, axis=1) > 0.0)[0])\n",
        "    self._data_matrix = self._data_matrix[nonzero_users, :]\n",
        "    self._effective_num_users = len(nonzero_users)\n",
        "\n",
        "    # Compute the SVD.\n",
        "    u, s, vh = np.linalg.svd(self._data_matrix, full_matrices=False)\n",
        "    # Keep only the largest singular values.\n",
        "    self._u_hat = u[:, :rank_k] * np.sqrt(s[:rank_k])\n",
        "    self._v_hat = np.transpose(\n",
        "        np.transpose(vh[:rank_k, :]) * np.sqrt(s[:rank_k]))\n",
        "    self._approx_ratings_matrix = np.matmul(self._u_hat, self._v_hat)\n",
        "\n",
        "    self._current_users = np.zeros(batch_size)\n",
        "    self._previous_users = np.zeros(batch_size)\n",
        "\n",
        "    self._optimal_action_table = np.argmax(\n",
        "        self._approx_ratings_matrix, axis=1)\n",
        "    self._optimal_reward_table = np.max(\n",
        "        self._approx_ratings_matrix, axis=1)\n",
        "    \n",
        "    self.reset()\n",
        "\n",
        "  @property\n",
        "  def batch_size(self):\n",
        "    return self._batch_size\n",
        "\n",
        "  @property\n",
        "  def best_action(self) -> int:\n",
        "    return np.argmax(np.sum(env._data_matrix, axis=0))\n",
        "\n",
        "  @property\n",
        "  def n_actions(self) -> int:\n",
        "    return self._data_matrix.shape[1]\n",
        "\n",
        "  def reset(self):\n",
        "    return self._observe()\n",
        "\n",
        "  def _observe(self) -> np.ndarray:\n",
        "    \"\"\"Returns the u vectors of a random sample of users.\"\"\"\n",
        "    sampled_users = random.sample(\n",
        "        range(self._effective_num_users), self._batch_size)\n",
        "    self._previous_users = self._current_users\n",
        "    self._current_users = sampled_users\n",
        "    batched_observations = self._u_hat[sampled_users]\n",
        "    return batched_observations\n",
        "\n",
        "  def step(self, action: List[int]) -> Tuple[int, float]:\n",
        "    \"\"\"Computes the reward for the input actions.\"\"\"\n",
        "    rewards = []\n",
        "    for i, j in zip(self._current_users, action):\n",
        "      rewards.append(self._approx_ratings_matrix[i, j])\n",
        "    self._observe()\n",
        "    return np.array(rewards)\n",
        "\n",
        "  def compute_optimal_action(self):\n",
        "    return self._optimal_action_table[self._previous_users]\n",
        "\n",
        "  def compute_optimal_reward(self):\n",
        "    return self._optimal_reward_table[self._previous_users]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9JzQKQgpMG-U"
      },
      "source": [
        "Now we are equipped to initialize our environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NtCnowoYPXZN"
      },
      "source": [
        "env = MovielensEnvironment('./ml-100k/u.data', rank_k=RANK_K, batch_size=1, num_movies=NUM_ACTIONS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HIdeM1t1MIID"
      },
      "source": [
        "Below we can check what this environment produces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K35v8fKoMLPz"
      },
      "source": [
        "env.reset()\n",
        "\n",
        "action = np.zeros(1, dtype=np.int32)\n",
        "reward = env.step(action)\n",
        "\n",
        "print(f'For users={env._previous_users}, we selected action={action} (optimal={env.compute_optimal_action()})')\n",
        "print(f'For users={env._previous_users}, we received reward={reward} (optimal={env.compute_optimal_reward()})')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C4zHqTdhPcKf"
      },
      "source": [
        "## Policy\n",
        "\n",
        "Let's see the selection's rate. The Upper Confidence Bound (UCB) algorithm's main idea is to use confidence intervals around current reward estimations; see picture below. The upper bound of the confidence interval ($\\hat{U}_t(a)$) is used for selecting the action (the true value is below with bound $Q(a) \\leq \\hat{Q}_t(a) + \\hat{U}_t(a)$ with high probability). In the early stages of run/training, we do not have much information about the reward, i.e., the large confidence interval.\n",
        "\n",
        "<center>\n",
        "  <img src=\"https://miro.medium.com/max/4800/1*p_4mvZ6r6ddbShd7tOT0sw.png\" alt=\"source: https://towardsdatascience.com/recommender-systems-using-linucb-a-contextual-multi-armed-bandit-approach-35a6f0eb6c4\" width=\"600\"/>\n",
        "</center>\n",
        "\n",
        "With UCB, $\\hat{a}_{t}$, the action chosen at time step $t$, is given by\n",
        "\n",
        "$$\n",
        "\\hat{a}_{t}^{\\ast} = argmax\\left(\\hat{Q}_{t}(a) + \\hat{U}_t(a)\\right) = argmax\\left( \\hat{Q}_{t}(a) + \\alpha \\sqrt{\\frac{\\log t}{N_{t}(a)}} \\right),\n",
        "$$\n",
        "\n",
        "where \n",
        "\n",
        "* $\\hat{Q}_{t}(a)$ is estimated value of action $a$ at time step $t$,\n",
        "* $N_{t}(a)$ is the number of times that action $a$ has been selected, prior to time $t$,\n",
        "* $\\alpha$ is a confidence value that controls the level of exploration.\n",
        "\n",
        "<br/>\n",
        "\n",
        "**TASK**: Add code for generating a batch of actions in the method `action`.\n",
        "\n",
        "> HINT: at the beggining $N_{t}(a)$ and $t$ is equal to $0$ and it couses division error. Try to add $1$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JyVSiknDPZUo"
      },
      "source": [
        "class Usb1Policy(object):\n",
        "  def __init__(\n",
        "      self, \n",
        "      values: List[float], \n",
        "      counts: List[int], \n",
        "      alpha: float = 2.0,\n",
        "      batch_size: int = 1\n",
        "    ):\n",
        "    self._counts = counts\n",
        "    self._values = values\n",
        "    self._alpha = alpha\n",
        "    self._batch_size = batch_size\n",
        "\n",
        "  def action(self) -> int:\n",
        "    # Hint: Add 1 to counts\n",
        "    Q = # YOUR CODE HERE\n",
        "    Na = # YOUR CODE HERE\n",
        "    alpha = # YOUR CODE HERE\n",
        "    t = # YOUR CODE HERE\n",
        "\n",
        "    # YOUR CODE GOES HERE\n",
        "    \n",
        "    # END OF YOUR CODE\n",
        "\n",
        "    return # YOUR CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7nGg2Zdma1kr"
      },
      "source": [
        "Below we can check what this policy produces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D6anif5CIgpn"
      },
      "source": [
        "p = Usb1Policy([0.0, 0.0, 0.0], [10, 1, 0], batch_size=2)\n",
        "print(f'USB1 policy - action: {p.action()}')\n",
        "\n",
        "# Expected output\n",
        "# USB1 policy - action: [2 2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L1zIu34hKdY2"
      },
      "source": [
        "## Agent\n",
        "\n",
        "As in previous implementation, the agent is responsible for updating policy parametrs. In case of UCB, it is mainly update $N_{t}(a)$ and $\\hat{Q}_t(a)$.\n",
        "\n",
        "<br/>\n",
        "\n",
        "**TASK**: Add code for updating important parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vj538Jq_KSzf"
      },
      "source": [
        "class Usb1Agent(object):\n",
        "  \n",
        "  def __init__(\n",
        "      self, \n",
        "      n: int, \n",
        "      alpha: float = 2.0,\n",
        "      batch_size: int = 1\n",
        "  ):\n",
        "    self._batch_size = batch_size\n",
        "    self._n = n\n",
        "\n",
        "    self.reset()\n",
        "\n",
        "    self.policy = Usb1Policy(\n",
        "        self._values, self._counts, alpha=alpha, batch_size=batch_size)\n",
        "\n",
        "  def reset(self):\n",
        "    self._counts = # YOUR CODE HERE  # N_t(a)\n",
        "    self._values = # YOUR CODE HERE  # Q_t(a)\n",
        "\n",
        "  def train(self, experience: Dict[str, float]):\n",
        "    \"\"\"Update policy parameters.\n",
        "\n",
        "    Args:\n",
        "      experience: dictionary with a single action and reward\n",
        "    \"\"\"\n",
        "    action = experience['action']\n",
        "    reward = experience['reward']\n",
        "\n",
        "    # YOUR CODE GOES HERE\n",
        "    \n",
        "    \n",
        "    # END OF YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U4hk-OkMa3LK"
      },
      "source": [
        "Below we can check how the training affects rewards estimation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yPgfe649K9R5"
      },
      "source": [
        "a = Usb1Agent(3, batch_size=1)\n",
        "\n",
        "experience = {'action': 2, 'reward': 1}\n",
        "a.train(experience)\n",
        "print(f'Q={a._values}')\n",
        "\n",
        "experience = {'action': 2, 'reward': 0}\n",
        "a.train(experience)\n",
        "print(f'Q={a._values}')\n",
        "\n",
        "# Expected output\n",
        "#Q=[0.0, 0.0, 1.0]\n",
        "#Q=[0.0, 0.0, 0.5]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M7_PdM45LZte"
      },
      "source": [
        "## Training\n",
        "\n",
        "Now we put together all the components that we introduced above: the environment, the policy, and the agent. We run the policy on the environment and output training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LRt6bId2LOws"
      },
      "source": [
        "def run(environment: object, agent: object, trials=100):\n",
        "  trajectory = []\n",
        "\n",
        "  experience = {'trial': 0, 'action': -1, 'observation': 0, 'reward': 0}\n",
        "\n",
        "  for i in range(trials):\n",
        "    experience['trial'] = i + 1\n",
        "    actions = agent.policy.action()\n",
        "    rewards = environment.step(actions)\n",
        "    optimal_rewards = environment.compute_optimal_reward()\n",
        "\n",
        "    for action, reward, optimal_reward in zip(actions, rewards, optimal_rewards):\n",
        "      experience['action'] = action\n",
        "      experience['reward'] = reward\n",
        "      experience['regret'] = optimal_reward - reward\n",
        "      \n",
        "      agent.train(experience)\n",
        "\n",
        "      trajectory.append(experience.copy())\n",
        "    \n",
        "  df_trajectory = pd.DataFrame.from_dict(trajectory)\n",
        "  df_cumsum = df_trajectory.groupby('action')['reward'].cumsum()\n",
        "  df_trajectory = df_trajectory.assign(cum_sum=df_trajectory['reward'].cumsum())\n",
        "  df_trajectory = df_trajectory.assign(action_cum_sum=df_cumsum)\n",
        "\n",
        "  return df_trajectory.astype({'action': 'int32', 'trial': 'int32'})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gLMqR8CrViNt"
      },
      "source": [
        "Down below is the code for creating all necessary instances. We have here a few parameters we can play with. `num_iterations` specifies how many times we run the trainer loop, `batch_size` defines how many actions are generated through one step and `alpha` controls the level of exploration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "toRds1GOLdzY"
      },
      "source": [
        "batch_size =   8 # @param {type:\"integer\"}\n",
        "num_iterations =   150 # @param {type:\"integer\"}\n",
        "alpha =   2.0 # @param {type:\"number\"}\n",
        "\n",
        "environment = MovielensEnvironment(\n",
        "    './ml-100k/u.data', \n",
        "    rank_k=RANK_K, \n",
        "    batch_size=batch_size, \n",
        "    num_movies=NUM_ACTIONS)\n",
        "step = environment.reset()\n",
        "\n",
        "agent = Usb1Agent(\n",
        "    environment.n_actions, \n",
        "    alpha=alpha, \n",
        "    batch_size=environment.batch_size)\n",
        "experience = {'action': [-1], 'reward': [0]}\n",
        "\n",
        "df_trajectory = run(\n",
        "    environment, agent, trials=num_iterations)\n",
        "\n",
        "print(f'Q={agent._values}')\n",
        "print(f'N={agent._counts}')\n",
        "print(f'best action={np.argmax(agent._values)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MSB0DGOkRM10"
      },
      "source": [
        "Now let's see the result. After running the last code snippet, the resulting plot (hopefully) shows that the average regret is going down as the agent is trained and the policy gets better in figuring out what the right action is, given the observation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HH5lCXfVRGxh"
      },
      "source": [
        "params = {\n",
        "    'algorithm': 'UCB1'\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ow4dq4DE3hdk"
      },
      "source": [
        "plot_regret(df_trajectory.groupby('trial').mean()['regret'], params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyCiqgRZKpub"
      },
      "source": [
        "Let's see the cumulative reward of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jwLLdWBWLt47"
      },
      "source": [
        "plot_cumsum(df_trajectory, params, show_actions=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6wd_tG1IPw9q"
      },
      "source": [
        "#### Multiple runs\n",
        "\n",
        "Because our environment selects users randomly, each run can produce different results. Let's what are the average results through multiple runs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DRMpP2hVLy4I"
      },
      "source": [
        "def experiment(\n",
        "    epochs: int = 1, trials: int = 10, alpha: float = 1.0, batch_size: int = 1):\n",
        "  trajectories = []\n",
        "  \n",
        "  environment = MovielensEnvironment(\n",
        "      './ml-100k/u.data', \n",
        "      rank_k=RANK_K, \n",
        "      batch_size=batch_size, \n",
        "      num_movies=NUM_ACTIONS)\n",
        "  params = {\n",
        "      'best_action': environment.best_action\n",
        "  }\n",
        "  \n",
        "  for epoch in trange(epochs):\n",
        "    step = environment.reset()\n",
        "    agent = Usb1Agent(\n",
        "        environment.n_actions, \n",
        "        alpha, \n",
        "        batch_size=environment.batch_size)\n",
        "    \n",
        "    df = run(environment, agent, trials=trials)\n",
        "    df['epoch'] = epoch + 1\n",
        "\n",
        "    trajectories.append(df)\n",
        "\n",
        "  df_trajectory = pd.concat(trajectories, ignore_index=True)\n",
        "\n",
        "  return df_trajectory.astype({'action': 'int32', 'trial': 'int32'}), params"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JmU80SrJW05i"
      },
      "source": [
        "Compared to a single run, we have one extra parameter. `epochs` controls the number of independent runs of the training loop."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ujsw0mHCP5-f"
      },
      "source": [
        "batch_size =   256# @param {type:\"integer\"}\n",
        "epochs = 25 # @param {type: \"integer\"}\n",
        "num_iterations =   150# @param {type:\"integer\"}\n",
        "alpha =   2.0 # @param {type:\"number\"}\n",
        "\n",
        "df_trajectory, params = experiment(\n",
        "    epochs=epochs, \n",
        "    trials=num_iterations, \n",
        "    alpha=alpha,\n",
        "    batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XomVaE1xMhOn"
      },
      "source": [
        "Now let's see the result. After running the last code snippet, the resulting plot (hopefully) shows that the average regret is going down as the agent is trained and the policy gets better in figuring out what the right action is, given the observation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MdeTF0i_Q_R8"
      },
      "source": [
        "params = {\n",
        "    'algorithm': 'UCB1'\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U5VgBKW83l6i"
      },
      "source": [
        "plot_regret(df_trajectory.groupby('trial').mean()['regret'], params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fucGaf7CKrts"
      },
      "source": [
        "Let's see the cumulative reward of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "td22YVlpP8_x"
      },
      "source": [
        "plot_cumsum(df_trajectory, params, show_actions=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "irJaPhGwKvVd"
      },
      "source": [
        "Let's see the selection's rate of the actions during the experiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E9PcddrqQ0ou"
      },
      "source": [
        "plot_actions(df_trajectory, params)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sC4K4TdMR3K7"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}