-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add generator of StreamingHandler into pipeline output. #4890
Comments
6 tasks
@yuanwu2017 @julian-risch I suggest we don't close this feature request but add it to the REST API task for V2 as a reminder that users ask for it |
@vblagoje sounds fair 👍 Will be good to have this use case in mind once we re-design the REST API. |
julian-risch
added
P3
Low priority, leave it in the backlog
2.x
Related to Haystack v2.0
labels
Jul 12, 2023
Closed
masci
changed the title
feat: add geneartor of StreamingHandler into pipeline output.
feat: add generator of StreamingHandler into pipeline output.
May 10, 2024
Blocked by deepset-ai/hayhooks#9 |
Won't fix in 1.x, let's use deepset-ai/hayhooks#9 to track the feature in 2.x |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
We are using the haystack to setup a streaming chatbot demo with LLM. But haystack doesn't have the streaming mode RestAPI.
We found that the streaming mode RestAPI needs a streaming generator to get the tokens from handler.
Describe the solution you'd like
Describe alternatives you've considered
Implement a stremer handler(customer Iterator object) in rest_api and pass it in pipeline.run. It needs to run entire pipeline in other thread or coroutine.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: