Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add generator of StreamingHandler into pipeline output. #4890

Closed
yuanwu2017 opened this issue May 12, 2023 · 4 comments
Closed

feat: add generator of StreamingHandler into pipeline output. #4890

yuanwu2017 opened this issue May 12, 2023 · 4 comments
Labels
2.x Related to Haystack v2.0 P3 Low priority, leave it in the backlog

Comments

@yuanwu2017
Copy link
Contributor

yuanwu2017 commented May 12, 2023

Is your feature request related to a problem? Please describe.
We are using the haystack to setup a streaming chatbot demo with LLM. But haystack doesn't have the streaming mode RestAPI.
We found that the streaming mode RestAPI needs a streaming generator to get the tokens from handler.

Describe the solution you'd like

  1. Add the streaming mode parameters in the promptNode run function, support the dynamically enabling and disabling.
  2. Add the TextIteratorStreamer object into pipeline output and run the pipe() of prompt node in new thread. It can simplify the implementation of streaming RestAPI and support the non-blocked mode.

Describe alternatives you've considered
Implement a stremer handler(customer Iterator object) in rest_api and pass it in pipeline.run. It needs to run entire pipeline in other thread or coroutine.

Additional context
Add any other context or screenshots about the feature request here.

@vblagoje
Copy link
Member

@yuanwu2017 @julian-risch I suggest we don't close this feature request but add it to the REST API task for V2 as a reminder that users ask for it

@julian-risch
Copy link
Member

@vblagoje sounds fair 👍 Will be good to have this use case in mind once we re-design the REST API.

@julian-risch julian-risch added P3 Low priority, leave it in the backlog 2.x Related to Haystack v2.0 labels Jul 12, 2023
@masci masci changed the title feat: add geneartor of StreamingHandler into pipeline output. feat: add generator of StreamingHandler into pipeline output. May 10, 2024
@masci
Copy link
Contributor

masci commented May 10, 2024

Blocked by deepset-ai/hayhooks#9

@masci
Copy link
Contributor

masci commented May 25, 2024

Won't fix in 1.x, let's use deepset-ai/hayhooks#9 to track the feature in 2.x

@masci masci closed this as completed May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 P3 Low priority, leave it in the backlog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants