Setting default alpha_value and fixing loading some newer DeepSeekCoder GGUFs #6111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ss_pos_emb for newer GGUFs 1. Ensures alpha_value resets to 1 after selecting a new model. Previously it didn't do it and it could potentially cause loading a newly selected model with sub-optimal settings. 2. Ensures rope scaling is loaded from GGUF metadata, notably from newer DeepSeekCoder finetunes where this value is saved in rope.scaling.factor instead of rope.scale_linear. Though there might be needed additional check for scaling type: linear or yarn. But I couldn't find information how to properly load Yarn models properly in TGW to decide if it's even needed to check the scaling type.
|
Your changes are correct and I wasn't aware that rope.scale_linear had been renamed to rope.scaling.factor. Nice catch. About scaling type, any metadata not explicitly passed while loading a model gets automatically detected by llama.cpp, so this is not an issue. |
PoetOnTheRun
pushed a commit
to PoetOnTheRun/text-generation-webui
that referenced
this pull request
Oct 22, 2024
…ss_pos_emb for newer GGUFs (oobabooga#6111)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist: