forked from intel-analytics/ipex-llm
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Chronos : add [how to tune a forecaster model] notebook (intel-analyt…
…ics#5399) * add how_to_tune_a_forecaster_model notebook * modify step1: data preparation * update based how to train * modify a bug of summary * update based on comment * move to howto * fix typo * add file * update based on comments * rename the tile
- Loading branch information
1 parent
0da86d0
commit eda0a35
Showing
3 changed files
with
308 additions
and
2 deletions.
There are no files selected for viewing
300 changes: 300 additions & 0 deletions
300
docs/readthedocs/source/doc/Chronos/Howto/how_to_tune_forecaster_model.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,300 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "[](https://colab.research.google.com/github/intel-analytics/BigDL/blob/main/docs/readthedocs/source/doc/Chronos/Howto/how_to_tune_forecaster_model.ipynb)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Tune forecaster on single node\n", | ||
| "\n", | ||
| "\n", | ||
| "## Introduction\n", | ||
| "\n", | ||
| "In this guidance, we demonstrate **how to tune forecaster on single node**. In tuning process, forecaster will find the best hyperparameter combination among user-defined search space, which is a common process if users pursue a forecaster with higher accuracy.\n", | ||
| "\n", | ||
| "Chronos support forecasting model's hyperparameter tuning in 2 sepaerated APIs (i.e. `Forecaster.tune` and `AutoTSEstimator`) for users with different demands:\n", | ||
| "\n", | ||
| "| |`Forecaster.tune`|`AutoTSEstimator`|\n", | ||
| "|-------------------|:---------------:|:---------------:|\n", | ||
| "|Single Node |✓ |✓ |\n", | ||
| "|Cluster |X |✓ |\n", | ||
| "|Performance-awared Tuning|✓ |X |\n", | ||
| "|Feature Selection |X |✓ |\n", | ||
| "|Customized Model |X |✓ |\n", | ||
| "\n", | ||
| "`Forecaster.tune` provides easier and more stright-forward API for users who are familiar with Chronos forecasters, it is recommened to try this method first.\n", | ||
| "\n", | ||
| "We will take `AutoformerForecaster` and nyc_taxi dataset as an example in this guide." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Setup\n", | ||
| "\n", | ||
| "Before we begin, we need to install chronos if it isn’t already available, we choose to use pytorch as deep learning backend." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!pip install --pre --upgrade bigdl-chronos[pytorch,automl]\n", | ||
| "!pip uninstall -y torchtext # uninstall torchtext to avoid version conflict\n", | ||
| "exit()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Data preparation\n", | ||
| "\n", | ||
| "First, we load the nyc taxi dataset.\n", | ||
| "\n", | ||
| "Currently, tune func only support **Numpy Ndarray input**." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from sklearn.preprocessing import StandardScaler\n", | ||
| "from bigdl.chronos.data.repo_dataset import get_public_dataset\n", | ||
| "\n", | ||
| "def get_tsdata():\n", | ||
| " name = 'nyc_taxi'\n", | ||
| " tsdata_train, tsdata_valid, _ = get_public_dataset(name)\n", | ||
| " stand_scaler = StandardScaler()\n", | ||
| " for tsdata in [tsdata_train, tsdata_valid]:\n", | ||
| " tsdata.impute(mode=\"linear\")\\\n", | ||
| " .scale(stand_scaler, fit=(tsdata is tsdata_train))\n", | ||
| " return tsdata_train, tsdata_valid\n", | ||
| "\n", | ||
| "tsdata_train, tsdata_valid = get_tsdata()\n", | ||
| "\n", | ||
| "input_feature_num = 1\n", | ||
| "output_feature_num = 1\n", | ||
| "lookback = 20\n", | ||
| "horizon = 1\n", | ||
| "label_len = 10\n", | ||
| "\n", | ||
| "train_data = tsdata_train.roll(lookback=lookback, horizon=horizon, label_len=label_len, time_enc=True).to_numpy()\n", | ||
| "val_data = tsdata_valid.roll(lookback=lookback, horizon=horizon, label_len=label_len,time_enc=True).to_numpy()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "train_data and val_data is compose of (x, y, x_enc, y_enc) as we set `time_enc=True` which is only necessary for Autoformer." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Tuning\n", | ||
| "\n", | ||
| "The first step of tuning a forecaster is to define forecaster with space parameters.\n", | ||
| "\n", | ||
| "There are several common space choices:\n", | ||
| "\n", | ||
| "`space.Categorical` : search space for hyperparameters which are categorical, e.g. a = space.Categorical('a', 'b', 'c', 'd')\n", | ||
| "\n", | ||
| "`space.Real` : search space for numeric hyperparameter that takes continuous values, e.g. learning_rate = space.Real(0.01, 0.1, log=True)\n", | ||
| "\n", | ||
| "`space.Int` : search space for numeric hyperparameter that takes integer values, e.g. range = space.Int(0, 100)\n", | ||
| "\n", | ||
| "How to change these hyperparameters might be tricky and highly based on experience, but lr, d_model, d_ff and layers or similar parameters usually has a great impact on performance." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import bigdl.nano.automl.hpo.space as space\n", | ||
| "from bigdl.chronos.forecaster.autoformer_forecaster import AutoformerForecaster\n", | ||
| "\n", | ||
| "autoformer = AutoformerForecaster(input_feature_num=input_feature_num,\n", | ||
| " output_feature_num=output_feature_num,\n", | ||
| " past_seq_len=lookback,\n", | ||
| " future_seq_len=horizon,\n", | ||
| " label_len=label_len,\n", | ||
| " seed=1024,\n", | ||
| " freq='t',\n", | ||
| " loss=\"mse\",\n", | ||
| " metrics=['mae', 'mse', 'mape'],\n", | ||
| " lr = space.Real(0.0001, 0.1, log=True),\n", | ||
| " d_model=space.Categorical(32, 64, 128, 256),\n", | ||
| " d_ff=space.Categorical(32, 64, 128, 256),\n", | ||
| " e_layers=space.Categorical(1,2),\n", | ||
| " n_head=space.Categorical(1,8))" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Then just call `tune` on the training data and validation data!\n", | ||
| "\n", | ||
| "In addition to data, there are three parameters which **need** to be specified : n_trials, target_metric and direction(or directions for multi-objective HPO).\n", | ||
| "\n", | ||
| "`n_trials`: number of trials to run. The more trials, the longer the running time, the better results.\n", | ||
| "\n", | ||
| "`target_metric`: the target metric to optimize, a string or an instance of torchmetrics.metric.Metric, default to 'mse'. If you want to try a multi-objective HPO, you need to pass in a list, for example ['mse', 'latency'] in which latency is a built-in metric for performance.\n", | ||
| "\n", | ||
| "`direction`: in which direction to optimize the target metric, \"maximize\" or \"minimize\", default to \"minimize\". If you want to try a multi-objective HPO, you need to set direction=None, and specify directions which is a list containing direction for each metric, for example ['minimize', 'minimize'].\n", | ||
| "\n", | ||
| "there are other two parameters which you **may** change their default values : epochs and batch_size." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "autoformer.tune(train_data, validation_data=val_data,\n", | ||
| " n_trials=10, target_metric='mse', direction=\"minimize\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Then, you can see the whole trial history by calling `search_summary()`." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "autoformer.search_summary()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "After `tune`, the model parameters of autoformer is **initialized** according to the best trial parameters. You need to fit the model again." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "autoformer.fit(train_data, epochs=4, batch_size=32)\n", | ||
| "# evaluate on val set\n", | ||
| "evaluate = autoformer.evaluate(val_data)\n", | ||
| "print(evaluate)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Save and load(Optional)\n", | ||
| "\n", | ||
| "After tuning and fitting, you can save your model by calling `save` with a filename." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "autoformer.save(checkpoint_file=\"best.ckpt\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Then, when you need to load the model weights, just call `load()` with corresponding filename." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "autoformer.load(checkpoint_file=\"best.ckpt\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Or if there is a new session, just define a new forecaster with **six necessary parameters: input_feature_num, output_feature_num, past_seq_len, future_seq_len, label_len, and freq**, then `load` with corresponding filename." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "new_autoformer = AutoformerForecaster(input_feature_num=input_feature_num,\n", | ||
| " output_feature_num=output_feature_num,\n", | ||
| " past_seq_len=lookback,\n", | ||
| " future_seq_len=horizon,\n", | ||
| " label_len=label_len,\n", | ||
| " freq='s')\n", | ||
| "new_autoformer.load(checkpoint_file=\"best.ckpt\")" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3.8.10 64-bit", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.8.10" | ||
| }, | ||
| "vscode": { | ||
| "interpreter": { | ||
| "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" | ||
| } | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters