Investigate Large Horizon Models

[antoinecarme]

Large Horizon Models (H large enough). Profiling for CPU/memory/speed.

Compute Prediction intervals for all tested models.
Use more sophistical forecast perf combination in model selection (mean ? max ?). decreasing time based weights ?
Take into account the shape of the prediction interval (esthetic for model precision).

[antoinecarme]

Large Horizon => Long Term ???

[antoinecarme]

Added movies of prediction intervals for Ozone and airline passengers models for increasing horizons (H=12, 24, 36, 48).

https://github.com/antoinecarme/PyAF_Benchmarks/tree/master/model_visualizer

[antoinecarme]

Add a voting system. Condorcet method by default

https://en.wikipedia.org/wiki/Condorcet_method

Each model has performance measures for each horizon. The value of the performance measure is considered as a horizon vote (H voters). Longer horizons are weighted (the longer the model performs, the better it is).

For the Condorcet method, each pair of models is compared (MAPE values).

In a pair competition , the winning model (smaller MAPE) is assigned a score of 1, this score is reweighted by the horizon length.

$$ Score(M) = \sum_h \sum_{m != M} h * \mathbb{1}_{MAPE(M , h) < MAPE(m, h)} $$

where $MAPE(m, h)$ is the h-point ahead MAPE for a model m on the validation dataset (multi-step error).

By design, the higher the voting score, the better it is.

[antoinecarme]

H = 10

generated with this script : https://github.com/antoinecarme/pyaf/blob/issue_213_Large_Horizon_Models/tests/long_term_forecasts/test_yosemite_temps_Horizon_10.py

[antoinecarme]

H = 50

[antoinecarme]

H = 100

[antoinecarme]

H = 200

[antoinecarme]

H = 500

[antoinecarme]

H = 800

[antoinecarme]

H = 1000

[antoinecarme]

TODO: Add some documentation about the new model selection procedure, with a detailed example.

For the moment :
#213 (comment)

[antoinecarme]

TODO: Keep some kind of backward compatibility. Use a new training option to choose between Condorcet and the old method.

Done.

5e496ad

[antoinecarme]

TODO : Do some "homework" about existing state of the art methods (FPP3 ?) !!!

[antoinecarme]

Advanced Review
Open Access
Review of automated time series forecasting pipelines
Stefan Meisenbacher, Marian Turowski, Kaleb Phipps, Martin Rätz, Dirk Müller, Veit Hagenmeyer, Ralf Mikut
First published: 09 August 2022
https://doi.org/10.1002/widm.1475

https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1475

[antoinecarme]

Fildes, Robert & Petropoulos, Fotios, 2015. "Simple versus complex selection rules for forecasting many time series," Journal of Business Research, Elsevier, vol. 68(8), pages 1692-1701.

https://www.sciencedirect.com/science/article/abs/pii/S0148296315001423

[antoinecarme]

Measuring forecast accuracy
Rob J Hyndman
Monday, 31 March 2014

https://robjhyndman.com/papers/forecast-accuracy.pdf

[antoinecarme]

Closing.