Does rtdl support multiple tokens?

[Up-in-the-wind]

Dear sir/madam：
There is a certain relationship between my different pieces of data. I would like to use each piece of data (array shape) as a token and input multiple tokens into FT-Transformer at the same time. How can I achieve this?

[Yura52]

Hi! Do I understand correctly that you have some data that is already embedded, and you don't need the "Feature Tokenizer" part ("FT-" in "FT-Transformer")? If yes, then, perhaps, rtdl.Transformer is what you are looking for? In its forward method, it accepts a generic tensor of the shape (batch_size, n_tokens, d_embedding). rtdl.FTTransformer uses rtdl.Transformer under the hood.

P.S. I hope that in the upcoming two weeks I will release a refactored version of FT-Transformer, perhaps, it will be easier to experiment with. With that release, rtdl.FTTransformer and rtdl.Transformer will be deprecated.

[Up-in-the-wind]

Hi, thanks you for the quick feedback and providing the solution ideas. I am also very looking forward to the update of rtdl.