🚨🚨 TextGenerationPipeline: rely on the tokenizer default kwargs

[gante]

What does this PR do?

Issue

TextGenerationPipeline.preprocess, where tokenization happens, has defined a few optional kwargs whose default value does not match the tokenizer defaults. This was causing:

1. A mismatch between the tokens resulting from tokenizer calls and the tokens pipeline was seeing, using default arguments on both
2. The issue seen in this thread -- add_special_tokens had to be manually set to True in the pipeline, despite it already being the default in the tokenizer.

(2 is a consequence of 1 :) )

This PR

🚨🚨 This PR changes the code to rely on the tokenizer's defaults when these flags are unset. This means some models using TextGenerationPipeline previously did not add a <bos> by default, which (negatively) impacted their performance. In practice, this is a breaking change.

Example of a script changed as a result of this PR:

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-9b-it", torch_dtype=torch.bfloat16, device_map="auto")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe("Foo bar"))

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[LysandreJik]

Thanks a lot @gante ! Let's keep it a bit on main to ensure it doesn't break too much the users' setups before releasing it.

[pcuenca]

Very nice, I agree it makes sense to make the behaviours consistent!