Changes to lmstudio to fix JSON decode error #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpacker
reviewed
Oct 31, 2023
| @@ -9,5 +9,8 @@ | |||
| # '\n#', | |||
| # '\n\n\n', | |||
| ], | |||
| "max_tokens": 500, | |||
| "max_tokens": 3072, | |||
| "lmstudio": {"context_overflow_policy": 2}, | |||
There was a problem hiding this comment.
Ideally we shouldn't really be using any context overflow policies, since MemGPT has its own way of handling this and this might conflict. But if this works as a bandaid fix for some other bug wrt open LLM integration I'm OK with leaving it for now.
There was a problem hiding this comment.
Ah ok. Well it may work ok without it. As you say, I think its the endpoint change that's doing the heavy lifting for... reasons lol.
…ack with recognizable error message) + add backwards compat option to use completions endpoint
mattzh72
pushed a commit
that referenced
this pull request
Oct 9, 2024
* Changes to lmstudio to fix JSON decode error * black formatting * properly handle context overflow error (propogate exception up the stack with recognizable error message) + add backwards compat option to use completions endpoint * set max tokens to 8k, comment out the overflow policy (use memgpt's overflow policy) * 8k not 3k --------- Co-authored-by: Matt Poff <[email protected]> Co-authored-by: cpacker <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @cpacker . I've made some tweaks to the LM Studio
settings.pyandapi.pycode which (for me at least) have resolved the issues using LM Studio as a back-end.http://localhost:1234/v1/chat/completionspromptJSON object withinmessageobject for compatability with endpoint and updated reference to response text accordingly.context_overflow_policyto option 2 (rolling context window)In local testing I'm getting expected behavior in chats and much improved performance. Might be worth checking the code in Windows for compatability (I don't have access at the moment).