Chat Stream Slow

[pascalrheaume]

Hi,

Upon upgrading to the latest git code, the chat stream has become notably slow, despite messages being fully generated and logged in the server console; evidently, as GPU usage remains at 0% after completion.

I attempted to eliminate the sleep command in private_gpt/ui/ui.py, although the existing sleep(0.02) duration is already quite brief. However, this modification did not yield any improvements. I've also experimented with various models (e.g., llama, mistral) and different local engines (ollama vs llama.ccp), but no improvement.

Any assistance would be greatly appreciated. If possible, I'd prefer to avoid reverting to an older version of PGPT to resolve this issue. Thank you!

[pascalrheaume]

Fresh restart and no slow streaming anymore. I didn't find a way to reproduce even after intensive summarize of documents via the recipe API. Closed the issue.