Implement Min P as a sampler option in HF loaders

[kalomaze]

Checklist:

• [✔️] I have read the Contributing guidelines.

A backport of the Min P sampler option that was merged into llama.cpp recently. I think this is a suitable alternative to accomodate for the flaws of both Top P (Top P doesn't work well if there's not good concentration of high probability for top tokens) and Top K (can limit diversity unnecessarily), and it has gained a very positive reception.

[kalomaze]

Technically the min_tokens_to_keep is not being properly followed here, even if that option usually goes unused, which probably needs to be changed before it gets merged.

[kalomaze]

Should be good to merge now.

EDIT: Somehow, Temperature is affecting how this chooses from the lists of considered tokens, but in llama.cpp, this would typically be measured and ran before Temperature which comes last, so the fact it's impacting the measurement means the HF samplers are coming after temp randomization, I believe.
That is potentially worrying since I assumed the standard was to apply temperature last? Though this might just be a design decision that hasn't been discussed. I know in koboldcpp you can customize a sampler order. Do we have that at all?

[oobabooga]

I was about to add min p manually since it looks cool. Thanks for the PR.

About the order, temperature does come before top p/top k in the transformers library:

https://github.com/huggingface/transformers/blob/acc394c4f5e1283c19783581790b3dc3105a3697/src/transformers/generation/utils.py#L825

I couldn't find any information as to how OpenAI does it.

Personally, I think that it makes sense to use temperature first.

[kalomaze]

I was about to add min p manually since it looks cool. Thanks for the PR.

About the order, temperature does come before top p/top k in the transformers library:

https://github.com/huggingface/transformers/blob/acc394c4f5e1283c19783581790b3dc3105a3697/src/transformers/generation/utils.py#L825

I couldn't find any information as to how OpenAI does it.

Personally, I think that it makes sense to use temperature first.

I can see arguments for both applying it last like how it is done in llama.cpp, and applying it like how it's done 'officially', so I think having this configurable / customizable to the end user would be helpful. Is that possible without changing how the transformers library functions?

Btw this is how it looks in GPT2, temp comes first there:

[kalomaze]

Changing how the sampler order logic works is out of scope for this PR though, so I'm ready for this to be merged in the meantime

[BadisG]

I think you should put the default min_p at 0 instead of 1, having it at 1 makes everything else useless because it's the equivalent to top_k = 1

[kalomaze]

I think you should put the default min_p at 0 instead of 1, having it at 1 makes everything else useless because it's the equivalent to top_k = 1

Is it not 0 by default? I thought I set it to that

[oobabooga]

Should be fixed now 4537853

[BadisG]

It's still at min_p = 1 by default for me

[oobabooga]

I messed up again, should be good after 7f9c1cb

[BadisG]

All right it's working now, thanks!

[BadisG]

I've also noticed that min_p and presence_penalty don't mix well, from time to time my generation stops abruptly because min_p removes some tokens and presence_penalty then removes the remaining tokens, leaving nothing to go on. I think this can be fixed by reversing the order of the samplers, presence_penalty should be applied before min_p. Maybe we should have the choice of sampler order, that would be a good feature to add, a bit like KoboldAI actually.

[kalomaze]

I've also noticed that min_p and presence_penalty don't mix well, from time to time my generation stops abruptly because min_p removes some tokens and presence_penalty then removes the remaining tokens, leaving nothing to go on. I think this can be fixed by reversing the order of the samplers, presence_penalty should be applied before min_p. Maybe we should have the choice of sampler order, that would be a good feature to add, a bit like KoboldAI actually.

Presence penalty... removes the tokens? I thought the rep pen (and associated variations of it) are biasing the logits and not actually truncating them, that's weird. And in koboldcpp rep pen comes first, so it being applied after the other truncation samplers is also foreign to me. I think the default kobold sampler order should be adapted + customizable

[ariez-xyz]

As far as I can tell, minP is missing from the generation parameters panel when llama.cpp is selected as a loader, though it works with llamacpp_HF. Could it be added to the llama.cpp settings as well?