fix: The LLM node setting with stream set to False.

[Hisir0909]

Checklist:

Important

Please review the checklist below before submitting your pull request.

• Please open an issue before creating a PR or link to an existing issue
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

Fixed the issue where the LLM node returned no results when the configured model did not use stream returns.
Fixes #8998

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Gemini works correctly with Stream = False.
• Gemini works correctly with stream = True.

[CXwudi]

The stream setting for Gemini in the workflow and chatflow works for me now. Since the change is modifying in the core part, I also tested with other providers like OpenAI, Arthorpic, and a few open sourced LLM from OpenRouter to make sure it doesn't produce regression.

The bug still exists in chatbot. I believe it is the same fix for chatbot, but it is totally fine if you just want to fix the bug in workflow and chatflow.

[Hisir0909]

The stream setting for Gemini in the workflow and chatflow works for me now. Since the change is modifying in the core part, I also tested with other providers like OpenAI, Arthorpic, and a few open sourced LLM from OpenRouter to make sure it doesn't produce regression.

The bug still exists in chatbot. I believe it is the same fix for chatbot, but it is totally fine if you just want to fix the bug in workflow and chatflow.

I've barely used this chatbot, so I've forgotten about it.😰

[Hisir0909]

In COT, there's no handling for the return values of non-stream modes. However, in fc, the stream mode is determined using stream_tool_call. I think the stream flag shouldn't be used in the chatbot; modifying the YAML description of stream is sufficient.

[CXwudi]

Sorry for the late reply. I agree that stream mode is not likely to be used in chatbot. Given that, user would usually expect the streaming behavior on the UI/UX side for chatbot. Hence, it is totally fine to not bother fixing it.

[Hisir0909]

@laipz8200 Why was it closed? We mean that fixing it under workflow and chatflow is correct; however, this stream setting should be disabled in agent mode, and the error can be ignored.

[laipz8200]

@laipz8200 Why was it closed? We mean that fixing it under workflow and chatflow is correct; however, this stream setting should be disabled in agent mode, and the error can be ignored.

Oops! Sorry for the misunderstanding.

[Hisir0909]

@laipz8200 Why was it closed? We mean that fixing it under workflow and chatflow is correct; however, this stream setting should be disabled in agent mode, and the error can be ignored.为什么会关闭？我们的意思是，在 workflow 和 chatflow 模式下进行修复是正确的；但是，在 agent 模式下应禁用此流设置，错误可以忽略。

Oops! Sorry for the misunderstanding.哎呀！对不起，让您误会了。

Thank you！

[laipz8200]

Our CI configuration has been updated, could you sync your code with the main branch to pass the CI?

[Hisir0909]

Our CI configuration has been updated, could you sync your code with the main branch to pass the CI?我们的 CI 配置已经更新，您能否将代码与主分支同步，以便通过 CI？

Sorry, I just saw it, I've synced now.

[hjlarry]

Hi @Hisir0909 @CXwudi , I haven't see this PR before, and already remove the stream config of yaml in this PR #9319 , caused by:

1. the error above
2. all the other models not have a stream config, the dify control whether stream mode itself

This PR seems to address the issue that the Gemini model's stream mode responds five times slower than when the stream mode is not enabled. The proposed solution is to let the user decide whether to activate the stream mode. However, I don't believe this is the right approach. The core issue is understanding why the Gemini model's stream mode is so slow. Stream mode should enhance UX, but currently, it detracts from it. This fundamental performance issue should be resolved by Google, not us.

[CXwudi]

2. all the other models not have a stream config, the dify control whether stream mode itself

If that's so, then I believe this PR is not needed.

This PR seems to address the issue that the Gemini model's stream mode responds five times slower than when the stream mode is not enabled. The proposed solution is to let the user decide whether to activate the stream mode. However, I don't believe this is the right approach. The core issue is understanding why the Gemini model's stream mode is so slow. Stream mode should enhance UX, but currently, it detracts from it. This fundamental performance issue should be resolved by Google, not us.

Uhhh, I think there is a misunderstanding here. We are not addressing performance issues, nor have we experienced any slowness from Gemini. This PR is mainly addressing #8998, which is a regression from #8678, as I mentioned in #8678 (comment).

[hjlarry]

Uhhh, I think there is a misunderstanding here. We are not addressing performance issues, nor have we experienced any slowness from Gemini. This PR is mainly addressing #8998, which is a regression from #8678, as I mentioned in #8678 (comment).

the performance issue is #8652 which seems #8678 to resolved.

[CXwudi]

I see, sorry that I wasn't aware of the root issue.

[Hisir0909]

Uhhh, I think there is a misunderstanding here. We are not addressing performance issues, nor have we experienced any slowness from Gemini. This PR is mainly addressing #8998, which is a regression from #8678, as I mentioned in #8678 (comment).

the performance issue is #8652 which seems #8678 to resolved.

I'd like to mention that I originally raised the performance issue #8652, and I'm willing to fix it. However, I didn't have time at that point, and someone else submitted PR #8678. Their approach simply added the stream flag in the YAML file without modifying the code. In DIFY, all LLMs are designed to handle both stream and non-stream processing, but the stream parameter is rarely used. This PR now adapts to the stream flag in the YAML file. If you think this is unnecessary or the approach is problematic, I will close the PR.

[hjlarry]

I'd like to mention that I originally raised the performance issue #8652, and I'm willing to fix it. However, I didn't have time at that point, and someone else submitted PR #8678. Their approach simply added the stream flag in the YAML file without modifying the code. In DIFY, all LLMs are designed to handle both stream and non-stream processing, but the stream parameter is rarely used. This PR now adapts to the stream flag in the YAML file. If you think this is unnecessary or the approach is problematic, I will close the PR.

Thank you for explaining the whole story from beginning to end. And the stream flag was already removed from all the YAML file, so this PR seems not so mush necessary, what do you think about this? @laipz8200

[laipz8200]

Thank you all for providing the context. If the performance issues with Gemini have been resolved, we won't need this PR anymore.

[Hisir0909]

Thank you all for providing the context. If the performance issues with Gemini have been resolved, we won't need this PR anymore.

Clearly, no. Only version 002 of Gemini-flash is normal because version 002 defaults to having the review interface turned off (according to Google). Other flash versions, including version 8B, are very slow in stream mode.

[laipz8200]

Since the steam option for the model has been removed, I think this PR is unnecessary. Let's wait for Gemini to fix this issue.