Question about look-back window selection

[tokaka22]

Hi, I just started learning time series prediction and I have an experimental question. In the FITS paper, it is mentioned that “Implementation details. We conduct grid search on the look-back window of 90, 180, 360, 720 and cutoff frequency, the only hyper-parameter.” So is the baselines' look-back window also the best one among 90, 180, 360, 720? Otherwise, different methods have different look-back windows, which seems unfair?

[VEWOXIC]

We do not think this is an unfair comparison since each model requires different input length. Equality does not mean fairness.
They simply force all the model to have the same input length 96 is an unfair comparison itself! For an example, the DLinear requires a long look back window to extract the periodicity. However, setting the input as 96 will greatly limit its performance! This is like you chop some professional runner's leg to match your height and then force him to have a running competition with you.

Further detail please check issue #6

[tokaka22]

I hope you don't mind my asking, but I'm currently conducting a related experiment and have a question about fairness. I understand what you mean, and I want to finally confirm that other baselines also do grid search on look-back window of 90, 180, 360, 720, is that right? Thank you~

[VEWOXIC]

Actually, seems not many people mention how they select the look-back window, which is weird. In our understanding, the length of look back window is also a hyper-parameter. It should be tuned according to your model design and dataset character. Most of transformer-based model use 96 as input because they cannot capture the periodicity, a longer look back window is not helpful (I think the paper of DLinear also report this finding).

Of course, it is good to conduct such grid search on all the baselines if you have fairness concern. However, considering the workload, we generally regard the developer of these models have report their result with their optimal input length.

[tokaka22]

Thank you for your kind reply. Generally speaking, if I directly quote the results of the baseline paper itself, then my proposed method performs a grid search on the look-back window, would reviewers typically have no concerns regarding this? (=。=... I would like to confirm if this rule is commonly accepted and practiced in the field of time series analysis, as you mentioned.) Thank you again~

[VEWOXIC]

As a reviewer, I can accept this protocol. However, I cannot guarantee that all the reviewers would have no concern. As I mentioned, not many people mention how they select the look-back window and they may have different point of view. Specifically, some of them may think that the input length is a predefined value of the specific task.

However, in our understanding and experience, the input length is never formally defined in any work and thus make it a tunable hyper-parameter. In this scenario, we should follow the commonly accepted rule -- comparing with the best performance of the baseline model. Further, we assume that authors have perform thorough hyper-parameter tuning (including the input length) before they report the final result. Thus, I think directly comparing your result with the ones that reported in baseline models' original paper is fair.

Overall, the input length is a tunable hyper-parameter is reasonable. Whether baseline models perform fine-tuning on it is their problem not ours. If you still have concern, you can just state this in your paper as a part of benchmark setting. I believe most of the reviewer with clear and constructive mind would accept it.

[tokaka22]

Thank you for your patient response!!!, which has clarified my understanding of experimental setups in the time series domain. I have indeed noticed that many papers do not provide details on the input length, necessitating a review of the code. Thank you once again for your assistance! =. =

[VEWOXIC]

Thanks for your interest and recognition of our work!