How to use PeriodicEmbeddings in rtdl.FTTransformer

[leiyu2023]

I found an implementation of PeriodicEmbeddings in this library, but I don't know how to use it in FFTransformer. This problem has been bothering me for a few days, can you give me some advice?@Yura52

[Yura52]

Hi @leiyu2023 !

Currently, it is not possible. The main branch contains many things (including PeriodicEmbeddings) that are work-in-progress and are not a part of the package that you get with pip install rtdl. Currently, I am not working on this project, and I don't know whether it will change.

If you are ready to accept the risks of using unfinished work, then you can take inspiration from this function and replace LinearEmbeddings with embeddings of your choice. IMPORTANTLY, you should not use pure PeriodicEmbeddings with Transformer, because PeriodicEmbeddings encodes only the "feature value" and does not encode the "feature index", which is crucial for Transformer-like models. How to solve this issue:

1. (simple solution) Use this function to create embeddings.
2. Add feature-wise trainable bias vectors similarly to how LinearEmbeddings does that.
3. etc.

[leiyu2023]

Thank you very much for your detailed answer, it's a good solution to my confusion