Usage examples for --logit-bias

[davidrpugh]

The logit bias option allows you to manually adjust the likelihood of specific tokens appearing in the generated text. By providing a token ID and a positive or negative bias value, you can increase or decrease the probability of that token being generated.

For example, when using the Gemma family of models, use --logit-bias 4521+1 to increase the likelihood of the token 'Hello', or --logit-bias 4521-1 to decrease its likelihood. Using a value of negative infinity, --logit-bias 15043-inf ensures that the token Hello is never produced. A more practical use case might be to prevent the generation of \code{begin} and \code{end} by setting the \ token (235286) to negative infinity with -l 29905-inf.

These usage examples depend on the tokenizer used by the model. The tokenizer-playground space on HF is an easy way to get token ids for commonly used tokenizers.