Error in Dify Framework with Qwen Model: Session Exceeds Input Length Limit

[BingFrost]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.71

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Initialize a conversation session using the Dify framework.
Conduct a series of consecutive question-answer interactions, ensuring that the cumulative text exceeds 6000 characters.

✔️ Expected Behavior

The framework should handle or truncate the input to comply with the model's limitations without causing a server error, allowing the conversation to proceed smoothly.

❌ Actual Behavior

An error is thrown, indicating that the input length has exceeded the acceptable range, resulting in a failed model invocation.

[dosubot]

@BingFrost Hello there! I'm here to help you with any bugs, questions, or contributions you have for the Dify framework. Just let me know how I can assist you while you wait for a human maintainer.

I found a similar issue that might help you. The issue titled "How do you limit the maximum token size of a session so that you don't have a huge token size if you keep talking in a session" was closed and last updated on August 27, 2024. The suggested solution is to limit the total number of tokens in the session if you are using the API ^[1].

To resolve the error in the Dify Framework with the Qwen Model where the session exceeds the input length limit, you can follow these steps:

1. Reduce the Prefix Prompt: Shorten the initial text or instructions provided to the model.
2. Shrink the Max Token: Decrease the maximum number of tokens the model is allowed to generate. The default maximum is 512 tokens, but you can adjust this value within the range of 1 to 4096 tokens.
3. Switch to a Model with a Larger Token Limit: If reducing the prefix prompt and shrinking the max token are not sufficient, consider switching to a language model with a larger token limit size. The Qwen2-72B-Instruct model has a context size of 32768 tokens, which is quite large ^{[2][3][4][5][6]}.

By adjusting these parameters, you can ensure that the input length stays within the model's acceptable range, thus preventing the error.

To continue talking to Dosu, mention @dosu.

[crazywoola]

What's the error message?