An error "context length exceeded" occurred during the conversation.

[StantonStetler]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.0

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

Create agent
Configure knowledge base
Debug dialogue chat on input box page
Error appears at the top of the page

✔️ Expected Behavior

Normal conversation

❌ Actual Behavior

Error occurred and no response

[dosubot]

Hello, @StantonStetler! I'm Dosu, a bot here to help you with bug reports, questions, and contributions. I'll assist you while we wait for a human maintainer.

I found some similar issues and discussions that might help resolve the "context length exceeded" error in your self-hosted Dify version 0.8.0:

1. Max Token Limits Incorrect? - This discussion addresses issues with max token limits and might provide insights into adjusting your settings ^[1].
2. Error in Dify Framework with Qwen Model: Session Exceeds Input Length Limit - This open issue suggests reducing the prefix prompt, shrinking the max token, or switching to a model with a larger token limit ^[2].
3. bug: max_token limit exceeded when change llm - This open issue recommends adjusting the max_tokens parameter to a maximum of 4096 tokens to resolve similar errors ^[3].

To resolve the "context length exceeded" error, you can lower the value of "Max token" in the parameter settings of the Prompt Eng. Reducing the "Max token" value will help manage the context length within the acceptable range. For example, setting max_tokens=500 ensures that only the first 500 tokens of the output are considered, discarding any excess ^[4][5].

Additionally, the AppRunner class in Dify's codebase provides methods to handle token calculations and adjustments. The get_pre_calculate_rest_tokens method calculates the remaining tokens available for the prompt by considering the model's context size and the max_tokens parameter. If the prompt tokens plus max_tokens exceed the model's context size, an InvokeBadRequestError is raised, indicating that the prompt or max_tokens needs to be reduced. The recalc_llm_max_tokens method recalculates the max_tokens if the sum of prompt tokens and max_tokens exceeds the model's token limit, ensuring it stays within the acceptable range ^[6].

By adjusting these settings, you should be able to avoid the "context length exceeded" error and ensure a smoother conversation flow.

To continue talking to Dosu, mention @dosu.

[crazywoola]

Try to adjust the max_tokens to a smaller one.

[StantonStetler]

Try to adjust the max_tokens to a smaller one.

The current max_token is the default 512, which is not very large, and in some cases, the length exceeds the limit. May I ask if this is due to the code? If so, which class should be modified?

[crazywoola]

Then I think the prompt is too long. \

Context = prompt + memory(2k) + max_tokens