Hyperparameter_Tutorial_Part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HDS-LEE Course on Hyperparameter Optimization - Part 2\n",
    "\n",
    "<img src='https://raw.githubusercontent.com/DLR-SC/Hyperparameter_tutorial/master/img/hds_lee_title.png' width=500px>\n",
    "\n",
    "This notebook summarizes several examples from the Talos GitHub repository. The original examples can be found\n",
    "on https://github.com/autonomio/talos/tree/master/examples\n",
    "\n",
    "Note: The output of this notebook is not reproducible. This means that **the final\n",
    "output changes with every usage of the code**. There are two main reasons for this\n",
    "stochastic behavior:\n",
    "* the traing set is quite small (400 samples) to reduce computing time (this is a tutorial so that\n",
    "there is a limited amount of time)\n",
    "* we use a 'random hyperparameter search strategy' and only cover a small percentage\n",
    "of parameter configurations (about 1%) to reduce computing time so that the output, of course, is\n",
    "not deterministic.\n",
    "\n",
    "In practice, this stochastic behavior should be strongly reduced since the number\n",
    "of training samples is usually much larger and the 'random hyperparameter search strategy'\n",
    "usually covers a larger percentage of hyperparameter configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "# required Talos __version__ = \"0.6.6\"\n",
    "import talos\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "The aim of the 2nd tutorial is the following:\n",
    "\n",
    "* Introduction of Talos\n",
    "* Use Talos for machine-assisted hyperparameter optimization of the Boston house pricing problem (1st Jupyter notebook).\n",
    "Also note that there is a large variety of hyperparameter tuning libraries and Talos is only\n",
    "one viable option (another recommend alternative is Keras Tuner)\n",
    "* Give general guidelines on hyperparameter optimization\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "##### 1. <a href=#one>Introduction of Talos</a>\n",
    "##### 2. <a href=#two>Use Talos for hyperparameter optimization </a>\n",
    "##### 3. <a href=#three>Guidelines on hyperparameter optimization</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction of Talos <a name=\"one\"></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Restore the original Keras code \n",
    "\n",
    "We start with the code that we have already used in the first notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from keras.datasets import boston_housing\n",
    "\n",
    "# load the data\n",
    "(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()\n",
    "\n",
    "# data normalization\n",
    "mean = train_data.mean(axis=0)\n",
    "train_data -= mean\n",
    "std = train_data.std(axis=0)\n",
    "train_data /= std\n",
    "\n",
    "test_data -= mean\n",
    "test_data /= std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Model preparation\n",
    "\n",
    "Talos works with any Keras model, without changing the structure of the model in any way.  \n",
    "The only difference in the Keras model is that a parameter is not set explicitly as before, e.g.\n",
    "\n",
    "<pre><code> model.add(layers.Dense(32, activation='relu'))    </code></pre>\n",
    "\n",
    "but instead taken from a dictionary, e.g.\n",
    "\n",
    "<pre><code>\n",
    "params = {'number_of_neurons' : [4, 6, 7, 8], 'activation' : ['relu'] }\n",
    "\n",
    "model.add(layers.Dense(params['number_of_neurons'], activation=params['activation']))    \n",
    "</code></pre>\n",
    "\n",
    "Afterwards, this dictionary and the model will be passed to Talos. In the dictionary we have \n",
    "three different ways to input values:\n",
    "\n",
    "- as stepped ranges (min, max, steps) if the parameter is a floating point number\n",
    "- as multiple values [in a list]\n",
    "- as a single value [in a list]\n",
    "\n",
    "For values we don't want to use, it's ok to set it as None."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tasks ## \n",
    "__Excercise 1:__\n",
    " - Create a Python dictionary that contains at least the following entries:\n",
    "    * 'number_of_layers' : 1, 2\n",
    "    * 'number_of_neurons' : 8, 16, 32, 64\n",
    "    * 'dropout_value' : None, 0.1, 0.2\n",
    "    * 'optimizer' : 'Adam', 'rmsprop'\n",
    "    * 'batch_size': 1, 2, 4, 8 \n",
    "    * 'epoch_number' : 10, 20, 40, 80\n",
    "    * anything that you want to modify further (e.g. learning_rate, activation_function, loss_function, ...)\n",
    " - What is the number of different hyperparameter configurations that is considered in the first six bullet points?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# parameter dictionary\n",
    "param = {\n",
    "        # enter your code here\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tasks ## \n",
    "__Excercise 2:__\n",
    " - Modify your original Keras model (Jupyter notebook 1) such that it uses uses the values from a \n",
    " dictionary 'p'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "# note: this build model works for a limited number of hidden layers, there are other options in Talos\n",
    "# if you are interested in a large number of layers (see: Outlook at the end of this tutorial)\n",
    "def build_better_model(x, y, val_data, val_targets, p):\n",
    "    model = models.Sequential()\n",
    "    \n",
    "    # replace the hyperparameter inputs with references to dictionary p \n",
    "    \n",
    "    # enter your code here\n",
    "        \n",
    "    model.add(layers.Dense(1))\n",
    "    model.compile(optimizer='Adam', loss='mse', metrics=['mae'])\n",
    "    \n",
    "    # make sure history object is returned by model.fit()\n",
    "    history = model.fit(x, y, \n",
    "                        # enter your code here\n",
    "                        #epochs=\n",
    "                        #batch_size= \n",
    "                        verbose=0)\n",
    "    \n",
    "    return history, model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Use Talos for hyperparameter optimization <a name=\"two\"></a> \n",
    "\n",
    "This part is quite simple. The Talos experiment just uses the <code> Scan()</code> command. In the following,\n",
    "we will investigate different arguments for this routine. However, there are only five necessary arguments\n",
    "that are required:\n",
    "* train_data (often known as 'x')\n",
    "* train_targets (often known as 'y')\n",
    "* params (the dictionary 'param' that we have created before)\n",
    "* model (the 'build_model' that we also have created before)\n",
    "* experiment_name (name of a folder in which the output, a csv-file, is stored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scan_object = talos.Scan(x=train_data,\n",
    "                         y=train_targets,\n",
    "                         model=build_better_model,\n",
    "                         experiment_name='find_optimal_params',\n",
    "                         params=param,\n",
    "                         round_limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If all different parameter configurations from the dictionary are used, the routine <code> Scan()</code> has to\n",
    "evaluate 768 different models. Depending on the neural network architecture and the number of training samples,\n",
    "this can be very time-consuming.\n",
    "\n",
    "As a solution, Talos offers several routines to limit the number of model configurations that is evaluated. Two useful\n",
    "commands of <code> Scan()</code> for this purpose are:\n",
    "\n",
    "* <code> round_limit=10</code> which limits the number of model evaluations to the specified integer value (e.g. \n",
    "10 model evaluations in this example).\n",
    "* <code> fraction_limit = 0.1</code> specifies the fraction of `params` that will be tested  (e.g. 10% of the \n",
    "configurations in this example).   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access the results through the <code>Scan</code> object "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Using <code> round_limit=10</code> we have investigated only ten parameter configurations (since the time\n",
    "for this tutorial is limited). But how do I get the ten configurations that have been used and what is their\n",
    "performance?\n",
    "\n",
    "For this purpose, the results of the scan object can be directly accessed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# accessing the results data frame\n",
    "scan_object.data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The output on my local computer is (the results should differ on your machine since you \n",
    "consider ten other random parameter configurations)\n",
    "\n",
    "<img src='https://raw.githubusercontent.com/DLR-SC/Hyperparameter_tutorial/master/img/talos_scan_output.png' \n",
    "width=1000px>\n",
    "\n",
    "There are several hyperparameter configurations with a mean average error (mae) in the range 2-3 \n",
    "(i.e. the average house pricing error is 2000-3000 dollar).\n",
    "Since the number of training samples is only 400, the results of the mean average error \n",
    "have a high variance and should not be overestimated. Nevertheless, we already obtain a certain \n",
    "understanding of a sensible size of some of the hyperparameters (e.g. 10 epochs is certainly not\n",
    " enough since 'mae' becomes large then).\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "You already noticed that your results differ from mine. This is due to the usage of a 'random search'\n",
    "strategy in Talos. More information on the search strategy are given \n",
    "in the summary details using <code>scan_object.details</code>.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# access the summary details\n",
    "scan_object.details\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "From the 768 possible hyperparameter configurations Talos considers a random subset\n",
    "of ten configurations (due to <code>round_limit=10</code>). This random subset \n",
    "is chosen with a 'Mersenne Twister' pseudorandom number generator. Other possible\n",
    "random choices in Talos are, for instance, 'Halton' or 'Sobol' quasi Monte Carlo sequences.\n",
    "\n",
    "Note: If there is no parameter such as  <code> round_limit=10</code> or\n",
    "<code> fraction_limit = 0.1</code> the default optimization strategy \n",
    "is called 'grid search'. This means that all hyperparameter permutations in \n",
    "a given dictionary are processed. In most cases, this is not recommended\n",
    "for anything but very small permutation spaces. \n",
    "Therefore, better use a 'random search' routine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to statistics and meta-data related with the Scan, the used data (x and y) \n",
    "together with the saved model and model weights for each hyperparameter permutation is stored in the Scan object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# accessing the saved models which returns a list of models\n",
    "scan_object.saved_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing the saved weights for models which returns a list of weights\n",
    "model_weights = scan_object.saved_weights\n",
    "\n",
    "# weights of first model \n",
    "model_weights[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Scan object can be further used, and is required, as input for Predict(), Evaluate(), and Deploy(). \n",
    "More about this in the corresponding sections below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing the Scan results with <code>Reporting()</code> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Scan process, the results are stored round-by-round in the corresponding experiment log which is a .csv file stored in the \n",
    "output directory (which is './find_optimal_params' our case). The Reporting() \n",
    "accepts as its source either a file name, or the Scan object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Scan object as input\n",
    "analyze_object = talos.Analyze(scan_object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# get the number of rounds in the Scan\n",
    "analyze_object.rounds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# get the lowest result for any metric (if lower is better)\n",
    "analyze_object.low('mae')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, we want to know the parameters of our best n=3 parameter runs. \n",
    "For this purpose we use <code> best_params</code>. It is alluring to consider the best \n",
    "result only (i.e. n=1 in the following cell) but keep in mind that due to the high\n",
    "variance in the result, a sequence of more models should be considered.\n",
    "\n",
    "The signature of <code> best_params()</code> is the following:\n",
    "* The first argument (here 'mae') is the metric that is considered.\n",
    "* The second argument is a list of metrics / loss functions that is not required here\n",
    "* The third argument gives the 'n' best results\n",
    "* Fourth argument: ascending | bool | Set to True when `metric` is to be minimized eg. loss or mae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# get the best n=3 paramaters\n",
    "analyze_object.best_params('mae', ['loss'], n=3, ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "It is very important to determine which of your hyperparameters has the largest impact on \n",
    "your performance metric. These are the parameters that should be further investigated. \n",
    "\n",
    "One solution to achieve this is to compute the linear correlation between your performance metric\n",
    "'mae' and your hyperparamters. The correlation coefficient is +1 in the case of a perfect direct\n",
    "(increasing) linear relationship, -1 in the case of a perfect decreasing (inverse) linear \n",
    "relationship and some values in (-1, 1) otherwise (indicating the degree of linear dependance\n",
    "between the variables).\n",
    "\n",
    "We employ <code> correlate()</code> for computing the correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# get correlation for hyperparameters against a performance metric such as 'mae' (we exclude 'loss' since \n",
    "# this is not a hyperparameter)\n",
    "analyze_object.correlate('mae', ['loss', 'round_epochs'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Since you have only then different hyperparameter runs, the output is almost random. However, in practice \n",
    "with hundreds or thousand of different configurations the result is much less stochastic noise in this output.\n",
    "\n",
    "<img src='https://raw.githubusercontent.com/DLR-SC/Hyperparameter_tutorial/master/img/talos_correlate.png' \n",
    "width=240px>\n",
    "\n",
    "In my special case, 'epoch_number' and 'number_of_neurons' have the strongest inverse effect on 'mae'\n",
    "(i.e. 'more epochs tend to lead to a lower mean average error'). The 'batch_size' seems to be less important. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also plot a heatmap of this correlation using <code> plot_corr()</code>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# heatmap correlation\n",
    "analyze_object.plot_corr('mae', ['loss', 'round_epochs'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The correlation between variables in one viable way to find the most important\n",
    "hyperparameters. A different approach is given by <code> plot_bars()</code>.\n",
    "Using this function, the metric 'mae' is plotted for different values of the \n",
    "hyperparameters 'number_of_neurons', 'number_of_layers' and 'epoch_number'. As mentioned\n",
    "before, 'mae' seems to decrease for larger values of these hyperparameters (in the\n",
    "specified region of the dictionary).\n",
    "\n",
    "The bar grid cotanins error bars if there is more than one result for the specified\n",
    "hyperparameter set ('number_of_neurons', 'epoch_number', 'number_of_layers'). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# a four dimensional bar grid\n",
    "analyze_object.plot_bars('number_of_neurons','mae', 'epoch_number', 'number_of_layers')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "My current output plot is\n",
    "\n",
    "<img src='https://raw.githubusercontent.com/DLR-SC/Hyperparameter_tutorial/master/img/talos_bar.png' \n",
    "width=1300px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tasks ## \n",
    "__Excercise 3:__\n",
    " - Apply the described process to your Talos output. In more detail:\n",
    "    * Find the hyperparameters that have the largest effect on the performance metric.\n",
    "    * Create a 2nd hyperparameter dictionary that further investigates these important hyperparameters\n",
    "    * Use <code> Scan() </code> to investigate the 2nd dictionary in more detail \n",
    "    (use larger values for round_limit than 10).\n",
    "    * Try to find a close-to-optimal model for this specific regression problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# your new hyperparameter dictionary\n",
    "param2 = {\n",
    "    # enter your code here\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "scan_object2 = talos.Scan(x=train_data,\n",
    "                         y=train_targets,\n",
    "                         model=build_better_model,\n",
    "                         experiment_name='find_optimal_params',\n",
    "                         # enter your code here\n",
    "                         #params=\n",
    "                         #round_limit=\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# if you want to see the full output of Scan()\n",
    "scan_object2.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Your best results are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# use Scan object as input\n",
    "analyze_object2 = talos.Analyze(scan_object2)\n",
    "\n",
    "# get the best n=3 paramaters\n",
    "analyze_object2.best_params('mae', ['loss'], n=3, ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# lowest mae with the parameters from analyze_object2.best_params('mae', ['loss'], n=1, ascending=True)\n",
    "analyze_object2.low('mae')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Deploy close-to-optimal model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Models with <code>Evaluate()</code> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models can be evaluated with <code>Evaluate()</code> against a k-fold cross-validation \n",
    "(<code>fold=K</code> specifies the number of repetitions in <code>Evaluate()</code>). \n",
    "Ideally at least 50% of the data, or more if possible, is kept completely out of the <code>Scan</code> process and only exposed into Evaluate once one or more candidate models have been identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_object = talos.Evaluate(scan_object2)\n",
    "\n",
    "# returns a list with 'folds=10' outputs\n",
    "all_mae_results =  evaluate_object.evaluate(test_data, test_targets, folds=10, metric='mae', task='continuous')\n",
    "\n",
    "# this is the average 'mae' that gives an aedequate estimation of our model error (on the test data)\n",
    "np.mean(all_mae_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a sufficiently performing model has been found, a deployment package can be easily created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploying Models with <code>Deploy()</code> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the right model or models have been found, you can create a deployment package with <code>Deploy()</code> which is then easy to transfer to a production or other environment, send via email, or upload to shared remote location. Best model is automatically chosen based on a given metric ('val_acc' by default).\n",
    "\n",
    "The Deploy package is a zip file that consist of: \n",
    "\n",
    "- details of the scan\n",
    "- model weights\n",
    "- model json\n",
    "- results of the experiment\n",
    "- sample of x data\n",
    "- sample of y data\n",
    "\n",
    "The <code>Deploy</code> package can be easily restored with <code>Restore()</code> which is covered in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creates a file waw_regression_deploy.zip (\\approx 13 KB) in the local folder \n",
    "# the parameter 'asc' has to be true if lower means better (e.g. loss, mae) but false otherwise (e.g. accuracy)\n",
    "talos.Deploy(scan_object=scan_object2, model_name='waw_regression_deploy', metric='mae', asc=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restoring Models with <code>Restore()</code>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "waw_regression = talos.Restore('waw_regression_deploy.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <code>Restore</code> object now consists of the assets from the Scan object originally associated with the experiment, together with the model that had been picked as 'best'. The model can be immediately used for making prediction, or use in any other other way Keras model objects can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# What is the 'best' model that we use for predictions?\n",
    "# this should be the same model as found with (better check this) \n",
    "# 'analyze_object2.best_params('mae', ['loss'], n=1, ascending=True)' from the previous cell \n",
    "waw_regression.model.get_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions with the model\n",
    "# if you are interested, compare the output with the ground truth 'test_targets'\n",
    "waw_regression.model.predict(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, for book keeping purpose, and for simplicity of sharing models with team members and other stakeholders, various attributes are included in the <code>Restore</code> object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the meta-data for the experiment\n",
    "waw_regression.details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the hyperparameter space boundary, these are the hyperparameter values that you have considered before\n",
    "waw_regression.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# these are the results from the different runs of Scan()\n",
    "waw_regression.results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That's almost all on Talos for today. Of course, there are several other routines that are provided by Talos. You find\n",
    "the most recent version (0.6.4 in November 2019) on GitHub (https://github.com/autonomio/talos).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Outlook: a large numbers of hidden layers in Talos\n",
    "\n",
    "You probably have noticed that our model architecture in 'def build_better_model()'\n",
    "can only be used for a relatively low number of hidden layers (since you have to add each layer manually).\n",
    "For a larger number of hidden layers, Talos provides a 'hidden_layers' model \n",
    "(https://github.com/autonomio/talos/blob/master/docs/Hidden_Layers.md). When 'hidden_layers' are used,\n",
    "several parameters must be included in your parameter dictionary. These parameters are \n",
    "'dropout', 'shapes', 'hidden_layers', 'first_neuron' and 'activation'. \n",
    "\n",
    "An application for our problem might look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "param = {'hidden_layers' : [1, 2],          # <--- required\n",
    "         'first_neuron' : [8, 16, 32, 64],  # <--- required\n",
    "         'dropout' : [0, 0.1, 0.2],         # <--- required\n",
    "         'shapes': ['brick'],               # <--- required\n",
    "         'activation': ['relu'],            # <--- required\n",
    "         'epoch_number' : [10, 20, 40, 80],\n",
    "         'optimizer' : ['Adam', 'rmsprop'],\n",
    "         'batch_size' : [1, 2, 4, 8]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "from talos.utils import hidden_layers\n",
    "\n",
    "def build_even_better_model(train_data, train_targets, val_data, val_targets, p):\n",
    "    model = models.Sequential()\n",
    "    \n",
    "    hidden_layers(model, p, 1) # <--- the required arguments are used here\n",
    "        \n",
    "    model.add(layers.Dense(1))\n",
    "    model.compile(optimizer='Adam', loss='mse', metrics=['mae'])\n",
    "    \n",
    "    # make sure history object is returned by model.fit()\n",
    "    history = model.fit(train_data, train_targets, epochs=p['epoch_number'], batch_size=p['batch_size'], verbose=0)\n",
    "    \n",
    "    return history, model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Bonus ## \n",
    "__Excercise 4:__\n",
    " - Use the feature 'hidden_layers' from Talos for a new hyperparameter search.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Guidelines on hyperparameter optimization <a name=\"three\"></a> \n",
    "\n",
    "* Hyperparameter optimization (at least as described in this tutorial with Talos) is an iterative process. You define the\n",
    "hyperparamter boundaries and start a search with <code>Scan()</code>. Using the results, you define new\n",
    "hyperparameter boundaries and a second search, ..., ..., final search. \n",
    "* In most situations, it is not necessary to perform a full grid search on all possible hyperparameter\n",
    "configurations (maybe in the order of thousands). Usually, it is sufficient to perform the scan on \n",
    "a random subset (10-30%).\n",
    "* It is important to determine those hyperparameter that have the largest influence on your performance\n",
    "metric. For this purpose, Talos provides routine such as <code>correlate()</code> assist you to find these hyperparameters.\n",
    "* All different hyperparameter combinations are evaluated on a validation set. As usual, only your final (after several iterations)\n",
    "model is evaluated with a test set that is independent of the validation set.\n",
    "* Even though Talos can be used analyze thousands of hyperparameter configurations, it is important to\n",
    "first get some kind of intuition/understanding which model architecture might be adequate (e.g. do you require \n",
    "fully connected neural networks?, convolutional neural networks?, what might be the rough number\n",
    "of hidden layers?). Otherwise, since the hyperparameter space grows exponentially you will not be able \n",
    "to evaluate a relevant subset of your high-dimensional hypercube within a reasonable amount of time\n",
    "(cf. Curse of Dimensionality).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thank you for your participiation! <br>\n",
    "We hope that you have enjoyed this tutorial."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}