Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[T5] Unused n_positions and max_position_embeddings. #8047

Closed
patrickvonplaten opened this issue Oct 26, 2020 · 0 comments · Fixed by #8518
Closed

[T5] Unused n_positions and max_position_embeddings. #8047

patrickvonplaten opened this issue Oct 26, 2020 · 0 comments · Fixed by #8518
Assignees

Comments

@patrickvonplaten
Copy link
Contributor

patrickvonplaten commented Oct 26, 2020

The T5Config has the parameter n_positions set to 512 and max_position_embeddigs referring to n_positions. However, neither max_position_embeddigs nor n_positions is used in the T5Model and T5 is not limited to max_position_embeddings. E.g.:

from transformers import T5Model
model = T5Model.from_pretrained("t5-small")

model.config.max_position_embeddings # shows 512

input_ids = torch.tensor([600 * [0]])  # input of size > 512

model(input_ids, decoder_input_ids=input_ids)  # works fine

I think we should delete the parameter.

@thomwolf - do you remember why we added max_position_embeddigs and n_positions to T5? The model does not seem to use these params and also should not be limited to 512 due to its relative position embeddings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants