IOH_Tutorial_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WRzrz6-DrW_O"
   },
   "source": [
    "This is the tutorial for the IOHexperimenter (python version). The full tutorial can also be found at: https://iohprofiler.github.io/IOHexp/python/\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UOtRDWfvsO1Z"
   },
   "source": [
    "## Basic installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uXFiXYKssKJG"
   },
   "source": [
    "This package can be installed directly from pip, using:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iE7KHCTUoVCo",
    "outputId": "d37f36d0-0a3a-4263-bd5b-61109f7b6b54"
   },
   "outputs": [],
   "source": [
    "pip install ioh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KNEcolWbsR3a"
   },
   "source": [
    "## Accessing problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0RPZbG7FoaTk"
   },
   "outputs": [],
   "source": [
    "from ioh import get_problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HO0Bl7kqrpaD"
   },
   "outputs": [],
   "source": [
    "#View docstring of get_problem\n",
    "?get_problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "opsk2tVZr5bG"
   },
   "source": [
    "Based on this, you can then access a problem, for example from the 'BBOB' suite of continuous problems:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9UxfuaA3rpcL"
   },
   "outputs": [],
   "source": [
    "#Create a problem object, either by giving the problem id from within the suite\n",
    "f = get_problem(7, dimension=5, instance=1, problem_type = 'BBOB')\n",
    "\n",
    "#Or by giving the problem name\n",
    "f2 = get_problem(\"Sphere\", dimension=5, instance=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iIoq0MMar7Ba"
   },
   "source": [
    "This problem contains a meta-data attributes, which consists of many standard properties, such as number_of_variables (dimension), name,..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AcFg9vTzrpeZ"
   },
   "outputs": [],
   "source": [
    "#Print some properties of the problem\n",
    "print(f.meta_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SuPDSjH1r_ZH"
   },
   "source": [
    "Additionally, the problem contains information on its bounds / conditions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KOTsuAqprpgh"
   },
   "outputs": [],
   "source": [
    "#Access the box-constrains for this problem\n",
    "f.constraint.lb, f.constraint.ub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wS7_3Su_sA89"
   },
   "source": [
    "The problem also tracks the current state of the optimization, e.g. number of evaluations done so far"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tzWtk553rpiu"
   },
   "outputs": [],
   "source": [
    "#Show the state of the optimization\n",
    "print(f.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v-fPF199sD0b"
   },
   "source": [
    "And of course, the problem can be evaluated easily:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BhLB0k0Yrpk6"
   },
   "outputs": [],
   "source": [
    "#Evaluate the problem\n",
    "f([0,0,0,0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wve-zVHesFta"
   },
   "source": [
    "## Running an algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QVDowejPsVdg"
   },
   "source": [
    "To show how to use IOHexperimenter to run an algorithm on a built-in problem, we can construct a simple random-search example wich accepts an IOHprofiler problem as its argument.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gXZfyp_Hrpm-"
   },
   "outputs": [],
   "source": [
    "#Create a basic random search algorithm\n",
    "import ioh\n",
    "import numpy as np\n",
    "\n",
    "def random_search(problem: ioh.problem.Real, seed: int = 42, budget: int = None) -> ioh.RealSolution:\n",
    "    np.random.seed(seed)\n",
    "    \n",
    "    if budget is None:\n",
    "        budget = int(problem.meta_data.n_variables * 1e4)\n",
    "\n",
    "    for _ in range(budget):\n",
    "        x = np.random.uniform(problem.constraint.lb, problem.constraint.ub)\n",
    "        \n",
    "        # problem automatically tracks the current best search point\n",
    "        f = problem(x)\n",
    "        \n",
    "    return problem.state.current_best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SHQ14JxHsXu5"
   },
   "source": [
    "To record data, we need to add a logger to the problem\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QOpmgadqsX3U"
   },
   "outputs": [],
   "source": [
    "#Import the ioh logger module\n",
    "from ioh import logger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jJ4hxQzhsX8S"
   },
   "source": [
    "Within IOHexperimenter, several types of logger are available. Here, we will focus on the default logger (called Analyzer as of version 0.32, Default for version 0.31 and earlier), as described [in this section](/data_format). Note that the logging can be customized by adding new triggers. Additionally, starting in version 0.32, the ability to store search points directly is added by using the store_positions-parameter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vPVrJNaVsYA2"
   },
   "outputs": [],
   "source": [
    "#Create default logger compatible with IOHanalyzer\n",
    "l = logger.Analyzer(root=\"data\", folder_name=\"run\", algorithm_name=\"random_search\", algorithm_info=\"test of IOHexperimenter in python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8c0zjAKLsYFa"
   },
   "source": [
    "This can then be attached to the problem\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Bm3TLMD7sYKJ"
   },
   "outputs": [],
   "source": [
    "#Add the logger to the problem\n",
    "f.attach_logger(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zGu4HchPsYO5"
   },
   "source": [
    "Now, we can run the algorithm. The logger will automatically store the relevant performance data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B-RSHZhosYTe"
   },
   "outputs": [],
   "source": [
    "random_search(f)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vpcQrcutsYat"
   },
   "source": [
    "For versions of ioh prior to 0.31, we need to explicitly ensure all data is written, so we should clear the logger after running our experiments. This is no longer be required after version 0.32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AhPDORl-sYgs"
   },
   "outputs": [],
   "source": [
    "l.flush() #Not needed for versions >= 0.32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WuparFpGsn8A"
   },
   "source": [
    "## Tracking algorithm parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SkmwvPzbsoKB"
   },
   "source": [
    "If we want to track adaptive parameters of the algorithm, we require an object in which the parameters of the algorithm are stored. In the below example, the random search algorithm is restructured into a class for this purpose. Alternatively, we could also create a seperate object which holds the parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ROp-hl4rsr1E"
   },
   "outputs": [],
   "source": [
    "class RandomSearch:\n",
    "    def __init__(self, budget: int):\n",
    "        self._budget = budget\n",
    "        self.seed = np.random.get_state()[1][0]\n",
    "        self._rng = np.random.default_rng(self.seed)\n",
    "\n",
    "    def __call__(self, func: ioh.problem.Real):\n",
    "        for i in range(self._budget):\n",
    "            x = self._rng.uniform(func.constraint.lb, func.constraint.ub)\n",
    "            f = func(x)\n",
    "        #Set new seed for future runs        \n",
    "        self.seed = np.random.get_state()[1][0]\n",
    "        self._rng = np.random.default_rng(self.seed)\n",
    "        return\n",
    "    \n",
    "    @property\n",
    "    def param_rate(self) -> int:\n",
    "        return np.random.randint(100)\n",
    "\n",
    "    \n",
    "#create an instance of this algorithm\n",
    "o = RandomSearch(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XU9ZZtsSss3x"
   },
   "source": [
    "We can then identify three different levels at which to track parameters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhpWO1O2ss-5"
   },
   "source": [
    "### Tracking adaptive parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2jbBYIGHs0Ag"
   },
   "source": [
    "The first type of parameters are the most common: parameters which we want to track during the search procedure, e.g. an adaptive stepsize. To track this type of parameter, we can make use of the 'watch' function of the logger as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NuvsFxyds1ll"
   },
   "outputs": [],
   "source": [
    "l.watch(o, ['param_rate'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oEri9Qt7s2I9"
   },
   "source": [
    "### Tracking run parameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CyjMvtaHs2LE"
   },
   "source": [
    "The second type of parameter is a per-run parameter. This can be something like the used random seed. To track this, we can use the following:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xqmQIF8As2sz"
   },
   "outputs": [],
   "source": [
    "l.add_run_attributes(o, ['seed'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Dzopu8fs21c"
   },
   "source": [
    "### Tracking experiment parameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MRO2BuT1s23f"
   },
   "source": [
    "The final type of parameters to track is the most high-level. This can be for example static algorithm parameters or other information about the experiment, which can be added as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-AAn7aL6s2-9"
   },
   "outputs": [],
   "source": [
    "l.add_experiment_attribute('budget', '1000')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L7FfVwJYtABK"
   },
   "source": [
    "Note: The methods for tracking parameters, e.g. `watch`, `add_run_attributes` and `add_experiment_attribute` can only be called before `f.attach_logger(l)` is called. Otherwise, the function will have no effect. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xf91pZ36tDiB"
   },
   "source": [
    "## Using the experimenter module\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WSn2ZFpitFSf"
   },
   "source": [
    "In addition to creating each problem individually, we can make use of the built-in experimenter module, which can be imported as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eSVTFTxStGXo"
   },
   "outputs": [],
   "source": [
    "from ioh import Experiment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0j8nUH00tpRI"
   },
   "outputs": [],
   "source": [
    "?Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NrYl1dovtJfE"
   },
   "source": [
    "At its core, the Experimenter object contains three parts: \n",
    "- An optimization algorithm (which takes a ioh-based problem as input)\n",
    "- Information on the collection of problems to be executed\n",
    "- Information on the logging procedure\n",
    "\n",
    "The suite object can be created using the suite-module from ioh as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uNCYueYhtLOu"
   },
   "outputs": [],
   "source": [
    "exp = Experiment(algorithm = o, #Set the optimization algorithm\n",
    "fids = [1,2,3], iids = [1,2,3,4,5], dims = [5,10], reps = 5, problem_type = 'BBOB', #Problem definitions\n",
    "njobs = 4, #Enable paralellization\n",
    "logged = True, folder_name = 'IOH_data', algorithm_name = 'Random_Search', store_positions = True, #Logging specifications\n",
    "experiment_attributes = {'budget' : '1000'}, run_attributes = ['seed'], logged_attributes = ['param_rate'], #Attribute tracking\n",
    "merge_output = True, zip_output = True, remove_data = True #Only keep data as a single zip-file\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "62bbdIHatL0O"
   },
   "source": [
    "This can be run as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AezL5TR3tME8"
   },
   "outputs": [],
   "source": [
    "exp.run()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6oKQ3HyktOW5"
   },
   "source": [
    "## Using custom problems\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7gwNcw5EtOZQ"
   },
   "source": [
    "In addition to the interfaces to the built-in problems, IOHexperimenter provides an easy way to wrap any problem into the same ioh-problem structure for easy use with the logging and experiment modules. This can be done using the 'wrap_real_problem' and 'wrap_integer_problem' problems. An example is shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JgqAoe7otSeJ"
   },
   "outputs": [],
   "source": [
    "from ioh import problem, OptimizationType\n",
    "\n",
    "#Define an evaluation method\n",
    "def f_custom(x):\n",
    "    return np.sum(x)\n",
    "#Call the wrapper\n",
    "problem.wrap_real_problem(f_custom, \"custom_name\",  optimization_type=OptimizationType.Minimization)\n",
    "\n",
    "#Call get_problem to instantiate a version of this problem\n",
    "f = get_problem('custom_name', instance=0, dimension=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y8HtUsIrtS_O"
   },
   "source": [
    "Note that you can also add transformations based on the instance id, for example as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oAwpGC3MtUki"
   },
   "outputs": [],
   "source": [
    "# Transformation function of x-attributes based on the instance id (numeric, default is 0). Note that argument order is fixed, but names are flexible.\n",
    "def transform_vars(x, instance):\n",
    "    x[1] += instance\n",
    "    return x\n",
    "\n",
    "# Transformation function of x-attributes based on the instance id (numeric, default is 0). Note that argument order is fixed, but names are flexible.\n",
    "def transform_obj(y, instance):\n",
    "    return y * instance\n",
    "\n",
    "# Function to calculate the objective (both x and corresponding objective value) based on the instance id (numeric, default is 0). Note that argument order is fixed, but names are flexible.\n",
    "def calc_obj(instance, dimension):\n",
    "    return [instance]*dimension, instance\n",
    "\n",
    "\n",
    "#We can then add these transformations when wrapping the problem:\n",
    "problem.wrap_real_problem(f_custom, name=\"custom_name2\",\n",
    "optimization_type=OptimizationType.Minimization, \n",
    "         transform_variables=transform_vars, transform_objectives=transform_obj, \n",
    "         calculate_objective=calc_obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wapu9lTftWzt"
   },
   "outputs": [],
   "source": [
    "f = get_problem('custom_name2', instance=3, dimension=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hkCvzsOptZoL"
   },
   "source": [
    "When using custom problems, they can be used with the Experiment class just the same as pre-defined problems. Note that you can see the problem id as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KLISBRzUta-6"
   },
   "outputs": [],
   "source": [
    "f.meta_data.problem_id\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Yf-r8_lWtcsw"
   },
   "outputs": [],
   "source": [
    "exp = Experiment(algorithm = o, #Set the optimization algorithm\n",
    "fids = [1,25], iids = [0], dims = [5,10], reps = 5, problem_type = 'BBOB', #Problem definitions\n",
    "njobs = 4, #Enable paralellization\n",
    "logged = True, folder_name = 'IOH_data', algorithm_name = 'Random_Search', store_positions = True, #Logging specifications\n",
    "experiment_attributes = {'budget' : '1000'}, run_attributes = ['seed'], logged_attributes = ['param_rate'], #Attribute tracking\n",
    "merge_output = True, zip_output = True, remove_data = True #Only keep data as a single zip-file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9AAMuWqytf59"
   },
   "source": [
    "Alternatively, we can use custom problems without first wrapping them, by using the 'add_custom_problem' function of Experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y_Lu7b3KtfB8"
   },
   "outputs": [],
   "source": [
    "exp = ioh.Experiment(0, fids=[], iids=[1], dims=[10], njobs=4)\n",
    "    exp.add_custom_problem(problem, \"problem\", \n",
    "         transform_variables=transform_vars, transform_objectives=transform_obj, calculate_objective=calc_obj)\n",
    "    exp.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SbUuEuzithyr"
   },
   "source": [
    "## Using the W-model problems\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TmE2KMXMth3J"
   },
   "source": [
    "In addition to the PBO and BBOB problems, the W-model problem generators (one based on OneMax and one based on LeadingOnes) are also avalable. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4HupVvm1tn3c"
   },
   "outputs": [],
   "source": [
    "?problem.WModelOneMax\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kMlrmSyOtlzv"
   },
   "outputs": [],
   "source": [
    "f = problem.WModelLeadingOnes(instance = 1, n_variables = 100, dummy_select_rate = 0.5, epistasis_block_size = 1, neutrality_mu = 0, ruggedness_gamma = 0 )"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "IOH_Tutorial.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}