-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support GPT4All Server API #11870
Comments
I also often get those errors/warnings in the commandline, when I try to send messages when connected to GPT4All server API. Not sure, if related.
|
Such an interesting issue and such an interesting behaviour... IMHO it's very wrong that there is no OpenAI API compatible mode in gpt4all. (Standardization made people's life much easier). However, I will look into this issue in more detail, because Gpt4All is a popular app and this is also important |
Ok... I reproduced it. I will try to fix it. |
Oh, @FeiLi-lab, @ThiloteE when you have the issue with truncated output, could you try to click on text area? Because there is some bug in the UI, when text are is not expanded. Could this be the case of truncated output? |
I will try to have a look at this on the weekend. |
Maybe could have been ggerganov/llama.cpp#9867 in upstream llama.cpp too. The fix would need some time to reach downstream GPT4All. |
I am also working on this issue. I think the problem that responses look like cutoff in the middle may come from the request. I ran the following two commands on my computer, one with max_token set and one without, and the result shows that the answer without max_token set was cutoff.
I set the max_token in the code, now the response looks complete. @ThiloteE I will refine the code later if you can review it. |
Oh, nice! Good to know. Yes, a pull-request would be nice, otherwise nobody can review. |
Welcome to the vibrant world of open-source development with JabRef! Newcomers, we're excited to have you on board. Start by exploring our Contributing guidelines, and don't forget to check out our workspace setup guidelines to get started smoothly. In case you encounter failing tests during development, please check our developer FAQs! Having any questions or issues? Feel free to ask here on GitHub. Need help setting up your local workspace? Join the conversation on JabRef's Gitter chat. And don't hesitate to open a (draft) pull request early on to show the direction it is heading towards. This way, you will receive valuable feedback. ⚠ Note that this issue will become unassigned if it isn't closed within 30 days. 🔧 A maintainer can also add the Happy coding! 🚀 |
…0' into Support-GPT4All-Server-API-JabRef#11870
Describtion of solution:
I want to use JabRef's AI feature locally. There are multiple applications out there that provide a server API. They very often offer an API that resembles OpenAI API.
GPT4All is such an application. Others are Llama.cpp, Ollama, LMStudio, Jan, KobolCPP. I am sure, there are more, but those are the most well known ones.
The grand advantage of those applications is that they offer more samplers, GPU acceleration, hardware support and support for models that have not been added to JabRef.
Problem
It kinda works with GPT4All already, but something is wrong. I believe the embeddings are not sent together with the prompt and responses look like they are cutoff in the middle.
GPT4All:
JabRef:
Additional context
JabRef preferences:
GPT4All preferences:
GPT4All model settings 1:
GPT4All model settings 2:
The text was updated successfully, but these errors were encountered: