Updated prompt_style to be moved to the main LLM settings and included various settings into the openailike mode. #1835
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Ms from llama_index can utilize this. I also included temperature, context window size, max_tokens, max_new_tokens into the openailike to help ensure the settings are consistent from the other implementations.
omartrigui
reviewed
Apr 11, 2024
|
@icsy7867 I think this is a very good and balanced solution, although I agree with the future steps. In order to merge this PR please remove the |
|
Also @icsy7867 please pull the latest changes of main into your branch. Otherwise CI tests won't work. |
|
@imartinez |
… instead of llamacpp.
imartinez
approved these changes
Apr 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I noticed, after trying new models and trying to implement new chat style templates, it did not seem to have an effect when using openailike.
I have moved the "prompt_style" under the LLM: section in the settings.yaml file as this should be able to be universally used across the various LLM modes.
However, since I am currently using VLLM to serve my model, I can test the other implementations. I have only changed the existing llamacpp and openailike to use the prompt_style (It is not currently being used anywhere else).
I also noticed that when using openailike, that I would easily exceed my various token limits. I believe this was due to max_tokens being set to none, or unlimited. I have changed this to use the "max_new_tokens" from the settings.yaml file.
I also included temperature, context_window,messages_to_prompt and completion_to_prompt values.
In the future, instead of hardcoding the different chat templates/styles, llama_index supports using .jinja formats:
https://github.com/vllm-project/vllm/tree/main/examples
https://github.com/chujiezheng/chat_templates/tree/main/chat_templates
Instead of changing the code, it would be more effective to have these templates in a directory, and to be able to reference them by their file name. That way if you want a new template, you could easily drop it into a specific folder and reference the filename, instead of having to change code. This allows that functionality to be dynamic without having to change anything within pgpt. However I dont have a PR for this yet.