When set num_beams in GenerationConfig, stop_strings parameter has no effect

[ZYM66]

System Info

uggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• transformers version: 4.46.2
• Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
• Python version: 3.10.15
• Huggingface_hub version: 0.26.2
• Safetensors version: 0.4.5
• Accelerate version: 1.1.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.5.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA GeForce RTX 4070 SUPER

Who can help?

@gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Code

from transformers import GenerationConfig, AutoTokenizer, AutoModelForCausalLM

generate_config = GenerationConfig(
    num_beams=3,
    do_sample=True,
    temperature=0.7,
    num_return_sequences=3,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.0,
    length_penalty=1.0,
    stop_strings=":",
    return_dict_in_generate=True,
    max_new_tokens=500,
    output_logits=True
)


tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(MODEL_PATH).cuda()

PROMPT = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"


tokens = tokenizer(PROMPT, return_tensors="pt").to(model.device)
out = model.generate(**tokens, generation_config=generate_config, tokenizer=tokenizer)

print(tokenizer.decode(out.sequences[0], skip_special_tokens=True))

Out

Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? To determine the total number of clips Natalia sold in April and May, we need to follow these steps:

1. Identify the number of clips sold in April.
2. Calculate the number of clips sold in May.
3. Add the number of clips sold in April and May together.

First, we know that Natalia sold 48 clips in April. Next, we need to find out how many clips she sold in May. According to the problem, she sold half as many clips in May as she did in April. Therefore, we calculate the number of clips sold in May as follows:
[
\text{Number of clips sold in May} = \frac{48}{2} = 24
]

Now, we add the number of clips sold in April and May together to find the total number of clips sold:
[
\text{Total number of clips sold} = 48 + 24 = 72
]

Thus, the total number of clips Natalia sold in April and May is \boxed{72}.

Expected behavior

when I set num_beams=1, the stop_strings works well !

[Rocketknight1]

@gante since this is a crossover stop strings / beam search issue, can you take a look? Feel free to ping me if you think it's an issue in the stop string handling, though, and I can try to fix it.