examples/02_model_hybrid/fm_deep_dive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Factorization Machine Deep Dive\n",
    "\n",
    "Factorization machine (FM) is one of the representative algorithms that are used for building hybrid recommenders model. The algorithm is powerful in terms of capturing the effects of not just the input features but also their interactions. The algorithm provides better generalization capability and expressiveness compared to other classic algorithms such as SVMs. The most recent research extends the basic FM algorithms by using deep learning techniques, which achieve remarkable improvement in a few practical use cases.\n",
    "\n",
    "This notebook presents a deep dive into the Factorization Machine algorithm, and demonstrates some best practices of using the contemporary FM implementations like [`xlearn`](https://github.com/aksnzhy/xlearn) for dealing with tasks like click-through rate prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 Factorization Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Factorization Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FM is an algorithm that uses factorization in prediction tasks with data set of high sparsity. The algorithm was original proposed in [\\[1\\]](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf). Traditionally, the algorithms such as SVM do not perform well in dealing with highly sparse data that is usually seen in many contemporary problems, e.g., click-through rate prediction, recommendation, etc. FM handles the problem by modeling not just first-order linear components for predicting the label, but also the cross-product of the feature variables in order to capture more generalized correlation between variables and label. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In certain occasions, the data that appears in recommendation problems, such as user, item, and feature vectors, can be encoded into a one-hot representation. Under this arrangement, classical algorithms like linear regression and SVM may suffer from the following problems:\n",
    "1. The feature vectors are highly sparse, and thus it makes it hard to optimize the parameters to fit the model efficienly\n",
    "2. Cross-product of features will be sparse as well, and this in turn, reduces the expressiveness of a model if it is designed to capture the high-order interactions between features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://recodatasets.z20.web.core.windows.net/images/fm_data.png?sanitize=true\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FM algorithm is designed to tackle the above two problems by factorizing latent vectors that model the low- and high-order components. The general idea of a FM model is expressed in the following equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\hat{y}(\\textbf{x})=w_{0}+\\sum^{n}_{i=1}w_{i}x_{i}+\\sum^{n}_{i=1}\\sum^{n}_{j=i+1}<\\textbf{v}_{i}, \\textbf{v}_{j}>x_{i}x_{j}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\hat{y}$ and $\\textbf{x}$ are the target to predict and input feature vectors, respectively. $w_{i}$ is the model parameters for the first-order component. $<\\textbf{v}_{i}, \\textbf{v}_{j}>$ is the dot product of two latent factors for the second-order interaction of feature variables, and it is defined as  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$<\\textbf{v}_{i}, \\textbf{v}_{j}>=\\sum^{k}_{f=1}v_{i,f}\\cdot v_{j,f}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to using fixed parameter for the high-order interaction components, using the factorized vectors increase generalization as well as expressiveness of the model. In addition to this, the computation complexity of the equation (above) is $O(kn)$ where $k$ and $n$ are the dimensionalities of the factorization vector and input feature vector, respectively (see [the paper](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf) for detailed discussion). In practice, usually a two-way FM model is used, i.e., only the second-order feature interactions are considered to favor computational efficiency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Field-Aware Factorization Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Field-aware factorization machine (FFM) is an extension to FM. It was originally introduced in [\\[2\\]](https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf). The advantage of FFM over FM is that, it uses different factorized latent factors for different groups of features. The \"group\" is called \"field\" in the context of FFM. Putting features into fields resolves the issue that the latent factors shared by features that intuitively represent different categories of information may not well generalize the correlation. \n",
    "\n",
    "Different from the formula for the 2-order cross product as can be seen above in the FM equation, in the FFM settings, the equation changes to "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\theta_{\\text{FFM}}(\\textbf{w}\\textbf{x})=\\sum^{n}_{j1=1}\\sum^{n}_{j2=j1+1}<\\textbf{v}_{j1,f2}, \\textbf{v}_{j2,f1}>x_{j1}x_{j2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $f_1$ and $f_2$ are the fields of $j_1$ and $j_2$, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to FM, the computational complexity increases to $O(n^2k)$. However, since the latent factors in FFM only need to learn the effect within the field, so the $k$ values in FFM is usually much smaller than that in FM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 FM/FFM extensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the recent years, FM/FFM extensions were proposed to enhance the model performance further. The new algorithms leverage the powerful deep learning neural network to improve the generalization capability of the original FM/FFM algorithms. Representatives of the such algorithms are summarized as below. Some of them are implemented and demonstrated in the microsoft/recommenders repository. \n",
    "\n",
    "|Algorithm|Notes|References|Example in Microsoft/Recommenders|\n",
    "|---------|-----|----------|---------------------------------|\n",
    "|DeepFM|Combination of FM and DNN where DNN handles high-order interactions|[\\[3\\]](https://arxiv.org/abs/1703.04247)|-|\n",
    "|xDeepFM|Combination of FM, DNN, and Compressed Interaction Network, for vectorized feature interactions|[\\[4\\]](https://dl.acm.org/citation.cfm?id=3220023)|[notebook](../00_quick_start/xdeepfm_criteo.ipynb) / [utilities](../../reco_utils/recommender/deeprec/models/xDeepFM.py)|\n",
    "|Factorization Machine Supported Neural Network|Use FM user/item weight vectors as input layers for DNN model|[\\[5\\]](https://link.springer.com/chapter/10.1007/978-3-319-30671-1_4)|-|\n",
    "|Product-based Neural Network|An additional product-wise layer between embedding layer and fully connected layer to improve expressiveness of interactions of features across fields|[\\[6\\]](https://ieeexplore.ieee.org/abstract/document/7837964)|-|\n",
    "|Neural Factorization Machines|Improve the factorization part of FM by using stacks of NN layers to improve non-linear expressiveness|[\\[7\\]](https://dl.acm.org/citation.cfm?id=3080777)|-|\n",
    "|Wide and deep|Combination of linear model (wide part) and deep neural network model (deep part) for memorisation and generalization|[\\[8\\]](https://dl.acm.org/citation.cfm?id=2988454)|[notebook](../00_quick_start/wide_deep_movielens.ipynb) / [utilities](../../reco_utils/recommender/wide_deep)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Factorization Machine Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Implementations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following table summarizes the implementations of FM/FFM. Some of them (e.g., xDeepFM and VW) are implemented and/or demonstrated in the microsoft/recommenders repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Implementation|Language|Notes|Examples in Microsoft/Recommenders|\n",
    "|-----------------|------------------|------------------|---------------------|\n",
    "|[libfm](https://github.com/srendle/libfm)|C++|Implementation of FM algorithm|-|\n",
    "|[libffm](https://github.com/ycjuan/libffm)|C++|Original implemenation of FFM algorithm. It is handy in model building, but does not support Python interface|-|\n",
    "|[xlearn](https://github.com/aksnzhy/xlearn)|C++ with Python interface|More computationally efficient compared to libffm without loss of modeling effectiveness|[notebook](fm_deep_dive.ipynb)|\n",
    "|[Vowpal Wabbit FM](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Matrix-factorization-example)|Online library with estimator API|Easy to use by calling API|[notebook](../02_model_content_based_filtering/vowpal_wabbit_deep_dive.ipynb) / [utilities](../../reco_utils/recommender/vowpal_wabbit)\n",
    "|[microsoft/recommenders xDeepFM](../../reco_utils/recommender/deeprec/models/xDeepFM.py)|Python|Support flexible interface with different configurations of FM and FM extensions, i.e., LR, FM, and/or CIN|[notebook](../00_quick_start/xdeepfm_criteo.ipynb) / [utilities](../../reco_utils/recommender/deeprec/models/xDeepFM.py)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other than `libfm` and `libffm`, all the other three can be used in a Python environment. \n",
    "\n",
    "* A deep dive of using Vowbal Wabbit for FM model can be found [here](../02_model_content_based_filtering/vowpal_wabbit_deep_dive.ipynb)\n",
    "* A quick start of Microsoft xDeepFM algorithm can be found [here](../00_quick_start/xdeepfm_criteo.ipynb). \n",
    "\n",
    "Therefore, in the example below, only code examples and best practices of using `xlearn` are presented."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 xlearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setups for using `xlearn`.\n",
    "\n",
    "1. `xlearn` is implemented in C++ and has Python bindings, so it can be directly installed as a Python package from PyPI. The installation of `xlearn` is enabled in the [Recommenders repo environment setup script](../../tools/generate_conda_file.py). One can follow the general setup steps to install the environment as required, in which `xlearn` is installed as well.\n",
    "2. NOTE `xlearn` may require some base libraries installed as prerequisites in the system, e.g., `cmake`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a succesful creation of the environment, one can load the packages to run `xlearn` in a Jupyter notebook or Python script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) \n",
      "[GCC 7.3.0]\n",
      "Xlearn version: 0.4.0\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import os\n",
    "import papermill as pm\n",
    "import scrapbook as sb\n",
    "from tempfile import TemporaryDirectory\n",
    "import xlearn as xl\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "%matplotlib notebook\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from reco_utils.common.constants import SEED\n",
    "from reco_utils.common.timer import Timer\n",
    "from reco_utils.recommender.deeprec.deeprec_utils import (\n",
    "    download_deeprec_resources, prepare_hparams\n",
    ")\n",
    "from reco_utils.recommender.deeprec.models.xDeepFM import XDeepFMModel\n",
    "from reco_utils.recommender.deeprec.io.iterator import FFMTextIterator\n",
    "from reco_utils.tuning.parameter_sweep import generate_param_grid\n",
    "from reco_utils.dataset.pandas_df_utils import LibffmConverter\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Xlearn version: {}\".format(xl.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the FM model building, data is usually represented in the libsvm data format. That is, `label feat1:val1 feat2:val2 ...`, where `label` is the target to predict, and `val` is the value to each feature `feat`.\n",
    "\n",
    "FFM algorithm requires data to be represented in the libffm format, where each vector is split into several fields with categorical/numerical features inside. That is, `label field1:feat1:val1 field2:feat2:val2 ...`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Microsoft/Recommenders utility functions, [a libffm converter](../../reco_utils/dataset/pandas_df_utils.py) is provided to achieve the transformation from a tabular feature vectors to the corresponding libffm representation. For example, the following shows how to transform the format of a synthesized data by using the module of `LibffmConverter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rating</th>\n",
       "      <th>field1</th>\n",
       "      <th>field2</th>\n",
       "      <th>field3</th>\n",
       "      <th>field4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1:1:1</td>\n",
       "      <td>2:4:3</td>\n",
       "      <td>3:5:1.0</td>\n",
       "      <td>4:6:1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1:2:1</td>\n",
       "      <td>2:4:4</td>\n",
       "      <td>3:5:2.0</td>\n",
       "      <td>4:7:1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1:3:1</td>\n",
       "      <td>2:4:5</td>\n",
       "      <td>3:5:3.0</td>\n",
       "      <td>4:8:1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1:3:1</td>\n",
       "      <td>2:4:6</td>\n",
       "      <td>3:5:4.0</td>\n",
       "      <td>4:9:1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1:3:1</td>\n",
       "      <td>2:4:7</td>\n",
       "      <td>3:5:5.0</td>\n",
       "      <td>4:10:1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   rating field1 field2   field3  field4\n",
       "0       1  1:1:1  2:4:3  3:5:1.0   4:6:1\n",
       "1       0  1:2:1  2:4:4  3:5:2.0   4:7:1\n",
       "2       0  1:3:1  2:4:5  3:5:3.0   4:8:1\n",
       "3       1  1:3:1  2:4:6  3:5:4.0   4:9:1\n",
       "4       1  1:3:1  2:4:7  3:5:5.0  4:10:1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_feature_original = pd.DataFrame({\n",
    "    'rating': [1, 0, 0, 1, 1],\n",
    "    'field1': ['xxx1', 'xxx2', 'xxx4', 'xxx4', 'xxx4'],\n",
    "    'field2': [3, 4, 5, 6, 7],\n",
    "    'field3': [1.0, 2.0, 3.0, 4.0, 5.0],\n",
    "    'field4': ['1', '2', '3', '4', '5']\n",
    "})\n",
    "\n",
    "converter = LibffmConverter().fit(df_feature_original, col_rating='rating')\n",
    "df_out = converter.transform(df_feature_original)\n",
    "df_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are in total 4 fields and 10 features.\n"
     ]
    }
   ],
   "source": [
    "print('There are in total {0} fields and {1} features.'.format(converter.field_count, converter.feature_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the use of `xlearn`, the following example uses the [Criteo data set](https://labs.criteo.com/category/dataset/), which has already been processed in the libffm format, for building and evaluating a FFM model built by using `xlearn`. Sometimes, it is important to know the total numbers of fields and features. When building a FFM model, `xlearn` can count these numbers automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "YAML_FILE_NAME = \"xDeepFM.yaml\"\n",
    "TRAIN_FILE_NAME = \"cretio_tiny_train\"\n",
    "VALID_FILE_NAME = \"cretio_tiny_valid\"\n",
    "TEST_FILE_NAME = \"cretio_tiny_test\"\n",
    "MODEL_FILE_NAME = \"model.out\"\n",
    "OUTPUT_FILE_NAME = \"output.txt\"\n",
    "\n",
    "LEARNING_RATE = 0.2\n",
    "LAMBDA = 0.002\n",
    "# The metrics for binary classification options are \"acc\", \"prec\", \"f1\" and \"auc\"\n",
    "# for regression, options are \"rmse\", \"mae\", \"mape\"\n",
    "METRIC = \"auc\" \n",
    "EPOCH = 10\n",
    "OPT_METHOD = \"sgd\" # options are \"sgd\", \"adagrad\" and \"ftrl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10.3k/10.3k [00:01<00:00, 8.67kKB/s]\n"
     ]
    }
   ],
   "source": [
    "tmpdir = TemporaryDirectory()\n",
    "\n",
    "data_path = tmpdir.name\n",
    "yaml_file = os.path.join(data_path, YAML_FILE_NAME)\n",
    "train_file = os.path.join(data_path, TRAIN_FILE_NAME)\n",
    "valid_file = os.path.join(data_path, VALID_FILE_NAME)\n",
    "test_file = os.path.join(data_path, TEST_FILE_NAME)\n",
    "model_file = os.path.join(data_path, MODEL_FILE_NAME)\n",
    "output_file = os.path.join(data_path, OUTPUT_FILE_NAME)\n",
    "\n",
    "if not os.path.exists(yaml_file):\n",
    "    download_deeprec_resources(r'https://recodatasets.z20.web.core.windows.net/deeprec/', data_path, 'xdeepfmresources.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following steps are from the [official documentation of `xlearn`](https://xlearn-doc.readthedocs.io/en/latest/index.html) for building a model. To begin with, we do not modify any training parameter values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE, if `xlearn` is run through command line, the training process can be displayed in the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training task\n",
    "ffm_model = xl.create_ffm()        # Use field-aware factorization machine (ffm)\n",
    "ffm_model.setTrain(train_file)     # Set the path of training dataset\n",
    "ffm_model.setValidate(valid_file)  # Set the path of validation dataset\n",
    "\n",
    "# Parameters:\n",
    "#  0. task: binary classification\n",
    "#  1. learning rate: 0.2\n",
    "#  2. regular lambda: 0.002\n",
    "#  3. evaluation metric: auc\n",
    "#  4. number of epochs: 10\n",
    "#  5. optimization method: sgd\n",
    "param = {\"task\":\"binary\", \n",
    "         \"lr\": LEARNING_RATE, \n",
    "         \"lambda\": LAMBDA, \n",
    "         \"metric\": METRIC,\n",
    "         \"epoch\": EPOCH,\n",
    "         \"opt\": OPT_METHOD\n",
    "        }\n",
    "\n",
    "# Start to train\n",
    "# The trained model will be stored in model.out\n",
    "with Timer() as time_train:\n",
    "    ffm_model.fit(param, model_file)\n",
    "\n",
    "# Prediction task\n",
    "ffm_model.setTest(test_file)  # Set the path of test dataset\n",
    "ffm_model.setSigmoid()        # Convert output to 0-1\n",
    "\n",
    "# Start to predict\n",
    "# The output result will be stored in output.txt\n",
    "with Timer() as time_predict:\n",
    "    ffm_model.predict(model_file, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output are the predicted labels (i.e., 1 or 0) for the testing data set. AUC score is calculated to evaluate the model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_file) as f:\n",
    "    predictions = f.readlines()\n",
    "\n",
    "with open(test_file) as f:\n",
    "    truths = f.readlines()\n",
    "\n",
    "truths = np.array([float(truth.split(' ')[0]) for truth in truths])\n",
    "predictions = np.array([float(prediction.strip('')) for prediction in predictions])\n",
    "\n",
    "auc_score = roc_auc_score(truths, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7498803439718372"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/papermill.record+json": {
       "auc_score": 0.7498803439718372
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sb.glue('auc_score', auc_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training takes 10.77s and predicting takes 0.93s.\n"
     ]
    }
   ],
   "source": [
    "print('Training takes {0:.2f}s and predicting takes {1:.2f}s.'.format(time_train.interval, time_predict.interval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the model building/scoring process is fast and the model performance is good. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Hyperparameter tuning of `xlearn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following presents a naive approach to tune the parameters of `xlearn`, which is using grid-search of parameter values to find the optimal combinations. It is worth noting that the original [FFM paper](https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf) gave some hints in terms of the impact of parameters on the sampled Criteo dataset. \n",
    "\n",
    "The following are the parameters that can be tuned in the `xlearn` implementation of FM/FFM algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Parameter|Description|Default value|Notes|\n",
    "|-------------|-----------------|------------------|-----------------|\n",
    "|`lr`|Learning rate|0.2|Higher learning rate helps fit a model more efficiently but may also result in overfitting.|\n",
    "|`lambda`|Regularization parameter|0.00002|The value needs to be selected empirically to avoid overfitting.|\n",
    "|`k`|Dimensionality of the latent factors|4|In FFM the effect of k is not that significant as the algorithm itself considers field where `k` can be small to capture the effect of features within each of the fields.|\n",
    "|`init`|Model initialization|0.66|-|\n",
    "|`epoch`|Number of epochs|10|Using a larger epoch size will help converge the model to its optimal point|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_dict = {\n",
    "    \"lr\": [0.0001, 0.001, 0.01],\n",
    "    \"lambda\": [0.001, 0.01, 0.1]\n",
    "}\n",
    "\n",
    "param_grid = generate_param_grid(param_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "auc_scores = []\n",
    "\n",
    "with Timer() as time_tune:\n",
    "    for param in param_grid:\n",
    "        ffm_model = xl.create_ffm()       \n",
    "        ffm_model.setTrain(train_file)     \n",
    "        ffm_model.setValidate(valid_file)\n",
    "        ffm_model.fit(param, model_file)\n",
    "\n",
    "        ffm_model.setTest(test_file)  \n",
    "        ffm_model.setSigmoid()        \n",
    "        ffm_model.predict(model_file, output_file)\n",
    "\n",
    "        with open(output_file) as f:\n",
    "            predictions = f.readlines()\n",
    "\n",
    "        with open(test_file) as f:\n",
    "            truths = f.readlines()\n",
    "\n",
    "        truths = np.array([float(truth.split(' ')[0]) for truth in truths])\n",
    "        predictions = np.array([float(prediction.strip('')) for prediction in predictions])\n",
    "\n",
    "        auc_scores.append(roc_auc_score(truths, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning by grid search takes 4.6 min\n"
     ]
    }
   ],
   "source": [
    "print('Tuning by grid search takes {0:.2} min'.format(time_tune.interval / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Lambda</th>\n",
       "      <th>0.001</th>\n",
       "      <th>0.010</th>\n",
       "      <th>0.100</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LR</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.5482</td>\n",
       "      <td>0.6122</td>\n",
       "      <td>0.7210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.5456</td>\n",
       "      <td>0.6101</td>\n",
       "      <td>0.7246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.5406</td>\n",
       "      <td>0.6147</td>\n",
       "      <td>0.7238</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Lambda   0.001   0.010   0.100\n",
       "LR                            \n",
       "0.0001  0.5482  0.6122  0.7210\n",
       "0.0010  0.5456  0.6101  0.7246\n",
       "0.0100  0.5406  0.6147  0.7238"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc_scores = [float('%.4f' % x) for x in auc_scores]\n",
    "auc_scores_array = np.reshape(auc_scores, (len(param_dict[\"lr\"]), len(param_dict[\"lambda\"]))) \n",
    "\n",
    "auc_df = pd.DataFrame(\n",
    "    data=auc_scores_array, \n",
    "    index=pd.Index(param_dict[\"lr\"], name=\"LR\"), \n",
    "    columns=pd.Index(param_dict[\"lambda\"], name=\"Lambda\")\n",
    ")\n",
    "auc_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f88c34559b0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAELCAYAAADawD2zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3gVVeLG8e9JCCC9Ewi95IKKCgioKEWQIkXFhiIqoqCiP5qIoiIqNlxcG7jKiouuAq5KUzoCKyqCoFK99JIQSOgEFkg5vz9mCAkpBMIkwLyf55mHmTNnzpm5Q95Mzp0711hrERGRi19IXu+AiIjkDgW+iIhPKPBFRHxCgS8i4hMKfBERn1Dgi4j4hAL/7LUDgsAG4JkM1j8IxAF/uNPDp6wvBkQDH6QquwdYCawAZgJl3PK3gL/c8klAiXNxAD53uvMHcBewBlgNfJmqfCawH/julPpfuG2uAsYCYW55N5xztwL4Gbgy57svJwQCgXaBQCAYCAQ2BAKBdOcyEAj8PRAI/OFO6wKBwH63/KpAIPBLIBBYHQgEVgQCgbtTbfOE254NBAJlTm3zQqXAPzuhwCigPXApTlBfmkG9icBV7vTPU9a9AixMtZwPeBdoCVyBEw5PuOvmAJe75euAZ8/FQfhYds5fbZzXuSlwGdAv1bq3gO4ZtPsFUAeoB1zCyV/ym4HmOOfvFeDjc3EQAoFAIN25DAQCac5lMBjsHwwGrwoGg1cB7wPfuquOAPcHg8HLcC4A3gkEAicupn4CWgNbc+Ewco0C/+w0xrky3AQcByYAt5zB9g2B8sDsVGXGnQq7/xYDdrjrZgOJ7vxioNLZ7rgA2Tt/j+AEyT53OTbVunnAoQzanQ5Yd1rCyfP0c6p2dP7OrcbAhmAwuCkYDGbnZ/EeYDxAMBhcFwwG17vzO3DOcVl3+fdgMLjFyx3PCwr8sxMBbE+1HOWWnep2nCv1r4HKblkIMBIYdErdBOAxnCGdHThXK59k0OZDwIyz3XEBsnf+It3pJ5yQbncG7Yfh/AUwM4N1PdH5O5ey+7NIIBCoClQHfshgXWMgP7DRg308b5jcfrSCMeZja22vTNb1AnoBXJK/bMMCYcVydd+y65bb2nNjqxvo+8QQAO7ueisNrr6CwU+9nFKnZKkSHI4/wvHjx+nR8x5u7XIzt3ToziO9u3PJJQV5750x3NOtC/Ub1OPpgS+RL18+vp48ln5PPs+WzdsYMfJFdu2KY+SI0SltDhz0GFc1qEf3ex7P9WM+E38rcU1e70KWqnVoTKUWV7BokDPKVuv2ppS5qiaLX/gspc5N/xpIcmISPzz6PoUrlKLjty/wbatnOH7wCADh19alXu+bmfPgyHTtXz+iJwlHjvHrsH+nKa9wXV2ue/VBvrvtFY7tj/fwCHOm++LBeb0L2TZr4c/8vPRPXnrqMQCmzVnIyr82MOTJnunqfjJ+Ert27023Lm7PPh4a8CLDBz/BlZdGplnX9t7HmPDhm5Qsfn5mUUbyV6pnMluXz4sOjTGlMlsF3JzZdtbaj3HHN0sWqXXePuRnR/ROIipVSFmuGBHOzpjYNHX27d2fMj/u04kMe/lpABo1voprr2tEz0e6UbhIIcLC8nM4/ghTpzgXg1s2bwNg8rfT6Tegd0obXe+9jTbtbuTWjhkNHcuZOBKzl8IVTv4XLRReiiM796WpczhmL3HLN2ATk4jfHseBjTEUqx7O7j83Zdl2/f63UbBUURYNHpumvGTdylw/4mFmdX/rvA77C035MqXZGbc7ZXlX3B7KlS6ZYd2ZC37iuf9Le+9E/OEj9BnyGk881DVd2F+MvBrSiQN+A5almn5zp3Ie9Zlrli9bQc2aValStRJhYWF0uaMDM6bPS1OnfPmyKfPtO7QiGHT+UuzVcyD16jbjysta8MKQN5g4fhIvvfgWMTt2EahTi9JlnCBqcWPTlG1atW5G3wG9uffu3vzvf0dz6SgvXnF/bqJY9XCKVC5LSFgoNW65hm1zlqeps3XWMipc57z3V6BkEYrVCOfQ1tiMmksReU8LIprXY/4ToyDVX86FK5am9Zh+LOz7Dw5u3nnuD8jHLq9Ti63RMUTF7CIhIYEZ83+ixXWN0tXbvD2ag4cOc+WlgZSyhIQE+r04gk5tmtO2+XW5udt5xpMrfJw3w1pZa7edusIYsz2D+heUpKQknh74Et9M/pTQ0FC++Pw//LV2Pc8+35c/lq9ixvR59H7sAdp1aEVSYiL79h2gz6NPZ9nmzp2xjHj9fb6f9SWJCYls37aDx91tRox8kQIF8jNp6r8A+G3pHwzoO9Trw7xo2aRkfnlhHO2+eBoTEsK6iQvZvy6aBk/dzu4/N7NtznKiF6ygUrN6dPnhTWxyMkuHj0+5Mu/wzQsUr1WBsMIF6br0PX58agzRC1fS9PUexEftptOUYQBsmbGUP96ZTP3+t1GgRBGue+1BAJITk5jaQefvXMgXGsqQJx/m0cHDSUpO5rb2N1KrWmU++HQClwVq0tIN/xk/LKJdy6YYc3K0Y+aCX1i2Yi37D8YzZdYCAIY/3Yc6tarzxbffM3biFPbs3c/tjwzkhsYNUoaNLmSejOEbY/oAi6y1f2aw7klr7funa+N8HtKRrJ3vY/iStQtpDF/Sy/UxfGvtqCzWnTbsRUTk3PNqSAdjTB2c+2EjcO5L3gFMtdau9apPERHJnCdv2hpjBuN8AMLgfABlqTs/3hiT2cfYRUTEQ15d4fcELrPWJqQuNMa8jfNckjc86ldERDLh1W2ZyUDFDMoruOtERCSXeXWF3w+YZ4xZz8mPPVcBanHygWAiIpKLvLpLZ6YxJhLnwUYROOP3UcBSa22SF32KiEjWPLtLh5NPDbQ4wzgn/hURkTzg1bN02gCjgfU4X/IBziNhaxljHrfWzs50YxER8YRXV/jvAq2ttVtSFxpjquM8M7yuR/2KiEgmvLpLJx/OmP2pojn5tW8iIpKLvLrCHwssNcZM4ORdOpWBrmT8pR4iIuIxr+7Sed0YMwXoDFzLybt0ullr13jRp4iIZM2zu3TcYF/jfhmKtdbuO902IiLiHa+epVPFGDPBGBML/AosMcbEumXVvOhTRESy5tWbthOBSUAFa21ta21tnMcqTMZ5qJqIiOQyrwK/jLV2YupP1Vprk6y1E4DSHvUpIiJZ8GoMf5kxZjQwjrR36TwA/O5RnyIikgWvAv9+nEckv8TJZ+lsB6ah2zJFRPKEV7dlHgc+dCcRETkPeDWGnyljTMfc7lNERPIg8IFGedCniIjv5cWXmL/oVZ8iIpI5fYm5iIhP6EvMRUR8Ql9iLiLiE/oScxERn9CXmIuI+ISXj0dOBhZ71b6IiJyZvLgPX0RE8oACX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QnPvsQ8p44nJ+b1LshZSjB5vQeSE6Zg4bzeBfGIrvBFRHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QkFvoiITyjwRUR8QoEvIuITCnwREZ9Q4IuI+IQCX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QkFvoiITyjwRUR8QoEvIuITCnwREZ9Q4IuI+IQCX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hP58noHLlQ33dScEW8NJTQ0lHH/msjIkR+mWX/ffXcw/NVniYnZBcA//jGOcf+amLK+aNEiLP99LlOnzmLggBcBmDFzAuHhZTl69BgAnTt1Jy5uDwBdunRgyHP9sNayauVaevTomxuHedGq3OIKmg7rjgkNYe34Bfwxelq6OjU7NqFh/y5gLXvWbmPek6MBuPnzpylfvyY7l65jRo+RKfWLVi5L61F9KFiiCHGrtvBD3w9JTkiiQpMA173YndJ1KzO3zwdsmr40147zYrRo8W+88c4/SEpO5vZO7Xi4+11p1r/57kcsWb4CgKPHjrF3335+mfU1f63byCt/+4D4w0cICQ2h1/1dad+6eZptX3t7NJOmz2Hp3EkpZTPn/ZfRY/+NwRCoXYMRwwZ7f5AeOW3gG2NCgZLW2t3ucn7gQaC/tbaut7t3fgoJCeHtv79Mp473ER29kx9/nMr338/hr782pKn3zTffpYT5qYYOHciiH39NV/7QQ/34ffnKNGU1a1bjqUGP07rV7ezff5CyZUufu4PxIRNiuH74A3x37xscjtlLl+9eZuucZexbvyOlTvFq5anfpxOTu7zE8QNHKFi6WMq6P//xPfkuyc+l3W5M0+41z3ZlxT9nsnHqYm54rQd1urZgzefziI/ew/wBH3Fl75tz7RgvVklJSQwfOYox77xGeLky3P1wX1pe34Sa1aum1Bnct3fK/Bf/mcLa9RsBKFiwAK+98BRVK0cQG7eHu3o+SdMmDSlWtAgAq9au42D84TT9bd0ezT8/n8jnH46keLGi7Nm3PxeO0jtZDukYY7oCe4EVxpiFxpiWwCagPdAtF/bvvHT11VexaeNWtmzZTkJCAl9/PY2OHdtke/ur6l9O2XJlmDfvx2zV79GjKx999Bn79x8ESLnql7NT7qqaHNyyi0Pb4khOSGLj1MVUa9MwTZ2697Zk1bi5HD9wBICjew6mrIv+aTUJ8UfTtVux6aVs+n4JAOu+/pHqbZ02D0XtZu9f28Farw7JN1auXUeVShWpHFGBsLAw2rdqzg8/Ls60/vS5C7m5dQsAqlWpRNXKEQCUK1uaUiVLsG//AcD5RTJy1CcMfLxnmu2/njqTrl06UbxYUQBKlyzhwVHlntON4T8PNLTWVgT6AzOBJ621t1lrl59Nh8aYHmez3fmkYsXyREWfvBqMjo6hQsXy6erdemt7fv11Bv/+YjQRERUAMMbw+uvP89yQ1zJs+6N/vMUvi6cz+JknU8pq1a5B7VrVmTvva+YvmMRNNzXPcFvJnsLhJYnfsTdlOT5mL4XDS6apU7xGOCVqhHPrt0O5bcowKre4Iss2C5YswvGDR7BJyZm2KTkXG7eb8HJlU5bLlytDbCYXQDt27iI6ZidNGl6Zbt3KNUESEhKp7P5cfvnNNFpefw1ly5RKU2/r9mi2bo/mvkcHcu8j/Vi0+LdzeDS573SBf9xauwHADfjN1tpJp9nmdF7KbIUxppcx5jdjzG+JiYdy2I13jDHpyuwpV2/Tp8+lbp3radKkPfPn/8SYMc5Yb6/e3Zk9az7R0THp2njoob40btyOm1rfSdPrGnHvvV0AyJcvlJq1qtOubVcefOBJRo1+g+LFi6XbXrIpw/OXdjkkNJTi1cOZeterzH1iFM1HPEz+YoVy1KbkXEavaQYvPQAz5i6kTYvrCQ0NTVMet3svz778FsOH9CckJITYuD3Mnv8j997ROV0biUlJbI2K5tMP3mTES8/w4hvvcPBQ/Lk4lDxxujH8csaYAamWi6Retta+ndFGxpgVmbRngPSXwifb+xj4GKBwoWrn7Y9LdPROKkVUTFmOiKjAzpjYNHX27j051vfp2PG88orzRk+Txg24rmkjHunVncKFC5E/fxiH448wdOibxOxw3uCNjz/MV19NpeHVV/Lll98SHb2TpUt+JzExka1bo1i/bhM1a1Vj+bLMXmbJyuGYvRSpePJKrkiFUhzZtS9NnfiYvcT+voHkxCQObY9j/8YYilcPJ+7PTRm2eXTvIfIXK4QJDcEmJWfYpuRc+XJl2Bkbl7K8K3Y3Zctk/J7WjLkLeW5gnzRl8YcP8/igoTzZ6wGuvNx5C3Lt+o1si4rh5rsfAuDo0WO0v+shZnw1lvJly3DlZXUIy5ePShXDqValElujoqlXN+DREXrrdFf4Y4CiqabUy0Wy2K48cD/QKYPpgh+AXrbsT2rWqkbVqpUICwvjjjs68f33c9LUCQ8/+Wdnh443EQw6bxw99FA/6gSacmnd63luyGt8+eW3DB36JqGhoZQu7QwB5MuXj3btb2TNmnUAfDdtNs2aXQtA6dIlqVW7Ols2b8uNQ70oxf65ieLVwilauSwhYaHU7HwNW+akHaHcMnsZFa+9FHCGa0rUCOfg1tiMmkux4+c11OjQGIDIO25gy+yzGvWULFxeJ5JtUTuI2rGThIQEZsxbSMvrr0lXb/PWKA4eiueqy0/eV5KQkEDfZ1+hc7tWtL3xhpTy5tc1ZuG0L5n9zThmfzOOggULMOOrsQC0anYtS5b/CcC+/QfYsj2ayhUreHyU3snyCt9am9XwS78sNv0OKGKt/SOD7RZke+/OU0lJSQwcMJQpUz8jNDSUzz77irVr1/P8C/1Zvnwl07+fy2OP9eDmDq1JSkxi77799O71VJZtFiiQnylTPyMsXz5CQkNZMP8nPh07HoA5cxbSqtUN/LZsDslJSTw35PU0f0HImbFJySx6YRwd/v00JjSE4MSF7FsXzdUDbyduxWa2zlnO9gUrqNSsHnfNexObnMwvr47n2H7nT/lbvnmBEjUrEFa4IPcteY8Fg8YQtXAli1+fwE2jnqDxoDvZvWoLaycsAKDslTVoO6YfBYoXomrr+lw94Ha+av1MHr4CF658+UIZ0v8xeg94nqSkJG7r2IZaNarywZjPuKxOJC1vcMJ/+twFtG/dPM3w68wffmTZH6vYf+AQk6fPBeDV5wZQJ7Jmpv01bdKQn5csp3O3XoSGhDKwT09KXMDDqebUsedsb2jMNmttlXO8PynO5yEdydrI0tfn9S5IDvT8/eW83gXJgbAyNTJ5VyNnH7zKtFEA4/xqbQxEABbYASyxZ/sbRkREciQngZ9pcBtj2gCjgfVAtFtcCahljHncWjs7B/2KiMhZyDLwjTGHyDjYDXBJFpu+C7S21m45pb3qwHTAl5/QFRHJS6d707ZoDtqNyqA8Ggg7yzZFRCQHvHp42lhgqTFmArDdLasMdAU+8ahPERHJgieBb6193RgzBegMXIszBBQFdLPWrvGiTxERyZpnj0d2g32NMaaUs2j1sUMRkTzkyRegGGOqGGMmGGNigV+BJcaYWLesmhd9iohI1rz6xquJwCSggrW2trW2NlABmAxM8KhPERHJgleBX8ZaO9Fam3SiwFqbZK2dAOjbO0RE8oBXY/jLjDGjgXGkvUvnAeB3j/oUEZEseBX49wM9cZ59H4Fzl852YBq6LVNEJE94dVvmceBDdxIRkfOAV2P4mTLGdMztPkVEJA8CH2iUB32KiPieZx+8MsbUAW4h7eORp1prX/SqTxERyZxXH7wajHO/vQGWAEvd+fHGGH3Vj4hIHvDqCr8ncJm1NiF1oTHmbWA18IZH/YqISCa8GsNPBipmUF7BXSciIrnMqyv8fsA8Y8x6Tn7wqgpQC3jCoz5FRCQLXt2HP9MYE8nJ77Q98XjkpakftyAiIrnHy8cjJwOLvWpfRETOTF7chy8iInlAgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QkFvoiITyjwRUR8QoEvIuITCnwREZ9Q4IuI+IQCX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hMKfBERn8iX1zuQmWOJCXm9C3KWiiTl9R5IjiTrBF6sdIUvIuITCnwREZ9Q4IuI+IQCX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QkFvoiITyjwRUR8QoEvIuITCnwREZ9Q4IuI+IQCX0TEJxT4IiI+ocAXEfEJBb6IiE8o8EVEfEKBLyLiEwp8ERGfUOCLiPiEAl9ExCcU+CIiPqHAFxHxCQW+iIhPKPBFRHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8QkFvoiITyjwRUR8QoEvIuITCnwREZ/Il9c7cKFq26YFb7/9MqEhIYz9dDwj3hqVZv393e/izTeeJ3rHTgBGj/6UsZ+OT1lftGgRVq1YwOQpM+nb73kAGtSvxyef/J1LChZkxswf6D9gaEr9Po/34PHHe5CYmMiMGfN45tlXc+EoL14VWlxBo1e6Y0JC2DB+Aas/mJauTpVOTbhiYBewln1rtvFTn9EA3PjF05RpUJPYJetY8MDIdNtdPfx+at7djIm1Hwag4bBulG96KQD5CuanYJlifFW3t4dHd3Fb9Osy3nj3Y5KSk7m9Yxsevu/ONOvffG8MS35fAcDRo8fYu/8Av8yYyF/rN/HKyFHEH/4fISEh9Lr/Ltq3agbAC2+8y+q/1mMtVKtckVeH9KdQoUuI2RXLkFf/zqH4wyQlJdP/0Qdodm2jXD/mc0WBfxZCQkJ4791XaXfzPURFxbD4l+lM+242a9euT1Pvq/9MTQnzU700bBD//XFxmrJRH7zOY48NZvGvy/hu6ue0a9uSmbPm06L5dXTu1Jb6DVpz/PhxypYt7dmx+YEJMTR+7QHmdX2DIzF7aT/9ZaJmLePA+h0pdYpWL8/lT3Zi9i0vcfzAEQqULpaybs2H3xN6SX5q33djurZLXVGd/MUKpSlbNuyLlPnAQzdR8vJq5/6gfCIpKYnhb3/ImL8PJ7xsae5+pD8tmzahZvUqKXUG/98jKfNffD2Ntes3AlCwQAFee24AVStHELt7D3f17EfTxg0oVrQIg598hCKFnfM24v0xfPntdzx83518NG4ibVveQNfbbmbj5m089vQwZv/nwg18DemchcaN6rNx4xY2b95GQkICX301hc6d2mZ7+wb161G+fFnmzPlvSll4eDmKFivK4l+XAfD5F1/TuXM7AHr3vp8Rb43i+PHjAMTF7TmHR+M/pevX5NCWXcRviyM5IYktUxZTqW3DNHVqdWvJun/N5fiBIwAc23MwZd3ORatJjD+arl0TYmjwwj38PnxCpn1Xu/Vatkz+5Rwdif+sXLuOKhEVqFwxnLCwMNq3asYPixZnWn/6vIXc3Lo5ANWqRFC1cgQA5cqUplTJ4uzbfwAgJeyttRw9dhxjDADGGA4fcf4PHDp8mLJlSnl2bLkh1wPfGFMnt/s81ypGhLM96uTVYFR0DBUrhqer1+W2m1m+bA4TJ3xMpUoVAec/0FsjhjL4meFp6kZUDCc6KiZlOToqhgi3zdq1a3D99Y35edE0fpj7NVc3vNKLw/KNQuElObJjb8rykZi9FKpQMk2dYjXCKVojnDZThtJ22jAqtLjitO1G9mhD1Ozl/C92f4brC0eUpkjlcuxatDpnB+BjsXF7CC9XNmW5fNkyxO7O+AJox85YonfsokmD9Odu5ZogCYmJVI6okFL2/Gvv0PyW7mzeFsW9t3cE4PEe9/Ld7Pm06vIAjw8axpB+j57jI8pl1tpcnYBtWazrBfzmTr1ye9/OYLrTWvvPVMvdrbXvn1KntLW2gDv/qLX2B3f+CWvt0+7xPWit/cAtb2StnZtq+xustdPc+VXW2vestcZa29hau9mdz+vX4UKdsnP+vrPWTrLWhllrq1tro6y1JU6sf+qpp/7m1jlRv6K1dpG1Np+7HJ9Bv4Mz6EfTGUyRkZF3RkZG/jPVcvfIyMgMX9PIyMjBGa0rWLDgoMjIyGBkZOQ1GWwTGhkZOToyMrKHuzwgMjJyoDt/bWRk5JrIyMiQvH4dznbyZAzfGPNeZquAEpltZ639GPjYi306x6KAyqmWKwE7TqmT+rJjDPCmO38tcENUVFR54BCQH4gH3nXbyajNKOBbwAJLgGSgDBCX0wPxqeycvyhgMZAAbAaCQG1gKcDatWs7AanftKkP1AI2uMuF3Plaqep0BfqckyPwr+ycuxPSvd6BQKBY+fLlhwEPBoPBdGNBwWAwKRAITAQGAZ8CPYF27rpfAoFAQZyfvdgcHkee8GpIpwewClh2yvQbcNyjPnPTUpwf/uo4gd0VmHpKnQqp5jsDa935bkCVSpUqrQSeAj4DngFicH4BXIPzi/F+YIq7zWTgxDuEkW6fu8/d4fhOds7fZKClO18G53XflEWb3wPhQDV3OkLasA8AJQEN4OfMUqB2IBCoHggEMjt3BAKBdK+3W3/SoUOH9gSDwf+kKjeBQKDWiXmgE/CXu3ob0MpdVxcoyAV8oeXVXTpLgVXW2p9PXWGMGeZRn7kpEXgCmAWEAmOB1cDLOL/UpgL/hxP0icBe4MFstPsY8C/gEmCGO+G2Pxbnl+hx4AGcq305O9k5f7OANsAaIAnniu/EX20/fv755zWAKjhXnD3d+lm5B5iAzluOBIPBxEAgkObcBYPB1YFA4GXgt2AweCL87wEmBIPB1K/3XUCzYsWKJQQCgT/csgeBFcC4QCBQDOdi60+cn0WAgcCYQCDQH+fcPXhKmxcUY+2533djTCngqLX2yDlv/CJhjOnlDmHJBUjn78Ll53PnSeCn6cAJf2ut3edpRyIikiVPxvCNMVWMMROMMXHAr8BSY0ysW1bNiz5FRCRrXr1pOxGYBIRba2tba2vhvIk5GWccU0REcplXgV/GWjvRWpt0osBam2StnQBcdM8FMMa0M8YEjTEbjDHPZLC+gDFmorv+19R/5RhjnnXLg8aYtqnKx7p/Fa3KnaMQOPtzaYwpbYyZb4yJN8Z8kNv7Lell41w2M8YsN8YkGmPuyIt9zG1eBf4yY8xoY0wTY0xFd2pijBkN/O5Rn3nCGBMKjALaA5cC9xhjLj2lWk9gn/uXzt9x78l363UFLsO513e02x44d+u08/wAJEVOziVwFHgB51ZbyWPZPJfbcO7S+TJ39y7veBX49wMrgZdwbp+aDQzDua2wu0d95pXGwFCul9MAAAO1SURBVAZr7SZr7XGcIatbTqlzCzDOnf8aaGWch3XcAkyw1h6z1m7G+aBOYwBr7X9xbueU3HPW59Jae9hauwgn+CXvnfZcWmu3WGtX4HyQ0Rc8uQ/ffYE/dKeLXQSwPdVyFNAkszrW2kRjzAGcoa0InE9zpt42wrtdldPIybnUB+HOL9k5l76TFw9P65jbfXrMZFB26r2umdXJzraSe3JyLuX8ovOUgbx4PPKF+zDpjGX3uSyVAYwx+YDiOMM1Z/JcEPFeTs6lnF/0s5UBzwLfGFPHGDPYGPOeMeZdd76utfZFr/rMI0uB2saY6saYzJ7tMRXncQgAdwA/WOcTb1OBru6dH9Vxnu+yJJf2W9LLybmU80t2zqXvePXBq8E4b5IYnABb6s6Pz+j2qAuZtTb1c1nWAl9Za1cbY142xnR2q30ClDbGbAAG4DwsDWvtauArnOe1zAT6nLiV1RgzHufBTwFjTJQxpmduHpcf5eRcAhhjtgBvAw+65+zUu0Ikl2TnXBpjGhljooA7gY+MMRf9FxV49SyddcBl1tqEU8rzA6uttbXPeaciIpIlr4Z0koGKGZRXwEe3QImInE+8ejxyP2CeMWY9J2+NqoLzfPAnPOpTRESy4NnTMo0xITgffojAGb+PApamftyCiIjkHs8fjywiIueHvLgPX0RE8oACX3zDGBPvQZtbjDFl8qJvkTOlwBcR8Qmv7tIRuSAYYzoBzwP5cb6kvJu1dpcxZhhQHedW4kicD1ldg/O43WigU6rPmQwyxrR05++11m5wPzn9Jc7P2MxU/RUBpgAlgTDgeWvtFG+PUsShK3zxu0XANdba+jifDn861bqaQAecx+r+G5hvra0H/M8tP+GgtbYx8AHwjlv2LvChtbYRsDNV3aPAbdbaBkBLYKT7qGwRzynwxe8qAbOMMSuBQThfRnPCDPcqfiUQyskr9ZVAtVT1xqf691p3vmmq8s9T1TXAa8aYFcBcnNuWy5+TIxE5DQW++N37wAfulXtvoGCqdccArLXJQEKqh6Qlk3Y41GZj/oRuQFmgobX2KmDXKX2KeEaBL35XHGdMHk4+BfNM3Z3q31/c+Z9wntAITsin7i/WWpvgjvtXPcs+Rc6Y3rQVPynkPh3xhLdxvnrzP8aYaJxvH6t+Fu0WMMb8inMBdY9b1hf40hjTF/gmVd0vgGnGmN+AP4C/zqI/kbOiT9qKiPiEhnRERHxCgS8i4hMKfBERn1Dgi4j4hAJfRMQnFPgiIj6hwBcR8Yn/B5WB+yFi76KKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "sns.heatmap(auc_df, cbar=False, annot=True, fmt=\".4g\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced tuning methods like Bayesian Optimization can be used for searching for the optimal model efficiently. The benefit of using, for example, `HyperDrive` from Azure Machine Learning Services, for tuning the parameters, is that, the tuning tasks can be distributed across nodes of a cluster and the optimization can be run concurrently to save the total cost.\n",
    "\n",
    "* Details about how to tune hyper parameters by using Azure Machine Learning Services can be found [here](https://github.com/microsoft/recommenders/tree/master/notebooks/04_model_select_and_optimize).\n",
    "* Note, to enable the tuning task on Azure Machine Learning Services by using HyperDrive, one needs a Docker image to containerize the environment where `xlearn` can be run. The Docker file provided [here](https://github.com/microsoft/recommenders/tree/master/docker) can be used for such purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmpdir.cleanup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='references'></a>\n",
    "1. Rendle, Steffen. \"Factorization machines.\" 2010 IEEE International Conference on Data Mining. IEEE, 2010.\n",
    "2. Juan, Yuchin, et al. \"Field-aware factorization machines for CTR prediction.\" Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 2016.\n",
    "3. Guo, Huifeng, et al. \"DeepFM: a factorization-machine based neural network for CTR prediction.\" arXiv preprint arXiv:1703.04247 (2017).\n",
    "4. Lian, Jianxun, et al. \"xdeepfm: Combining explicit and implicit feature interactions for recommender systems.\" Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2018.\n",
    "5. Qu, Yanru, et al. \"Product-based neural networks for user response prediction.\" 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016.\n",
    "6. Zhang, Weinan, Tianming Du, and Jun Wang. \"Deep learning over multi-field categorical data.\" European conference on information retrieval. Springer, Cham, 2016.\n",
    "7. He, Xiangnan, and Tat-Seng Chua. \"Neural factorization machines for sparse predictive analytics.\" Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 2017.\n",
    "8. Cheng, Heng-Tze, et al. \"Wide & deep learning for recommender systems.\" Proceedings of the 1st workshop on deep learning for recommender systems. ACM, 2016.\n",
    "9. Langford, John, Lihong Li, and Alex Strehl. \"Vowpal wabbit online learning project.\" (2007)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_base)",
   "language": "python",
   "name": "reco_base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}