[Usage]: Specify the number of tokens to be generated

[oximi123]

Your current environment

vllm

How would you like to use vllm

I want to specify the number of generated tokens for each request using vllm. Is there a way to do so?

[simon-mo]

You can use a combination of max_tokens and ignore_eos in sampling params to achieve this. https://docs.vllm.ai/en/latest/dev/sampling_params.html

There's also active work in #3124 for minimal tokens.