You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The SentenceTransformer models have a max_seq_len attribute.
In theory, we could set it with the model_max_length in tokenizer_kwargs which then eventually should set that attribute here.
However, it seems to be unreliable. We saw it with bge-m3 but could also be the case for other models. (Colab)
This can result in OOM error when embedding.
Update: found the "issue". The max_seq_length is read into the kwargs from this config json at this point in _load_sbert_model and thus the max_seq_length is not None here and so it doesn't use model_max_length from the tokenizer_kwargs.
Describe the solution you'd like
Would be good to have the ability to set the max_seq_len in the (currently three) SentenceTransformers components as in v1.
We could possibly also intercept the tokenizer_kwargs and use model_max_length from it if it's set.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The SentenceTransformer models have a max_seq_len attribute.
In theory, we could set it with the
model_max_length
in tokenizer_kwargs which then eventually should set that attribute here.However, it seems to be unreliable. We saw it with bge-m3 but could also be the case for other models. (Colab)
This can result in OOM error when embedding.
Update: found the "issue". The
max_seq_length
is read into the kwargs from this config json at this point in _load_sbert_model and thus the max_seq_length is not None here and so it doesn't usemodel_max_length
from the tokenizer_kwargs.Describe the solution you'd like
Would be good to have the ability to set the
max_seq_len
in the (currently three) SentenceTransformers components as in v1.We could possibly also intercept the tokenizer_kwargs and use
model_max_length
from it if it's set.The text was updated successfully, but these errors were encountered: