Use response synthesizer in context chat engines

[logan-markewich]

The context chat engines do not handle token overflows well. It requires the user to

• limit the token limit on their memory
• limit the overall top-k on each retrieval

By using a response synthesizer, we can make this a little more painless.

This needed up being a decent refactor, but I think it simplified a lot too

[logan-markewich]

(Ignore the failing tests, its just lancedb being flakey)

[enrico-stauss]

Resolved by d399209

[shadyelgewily-slimstock]

Hi @logan-markewich, we just noticed this (seemingly) behavioral change by upgrading llama-index to the latest version. In previous versions of this class, there was no "refine" prompt. Is the additional refine prompt essential to preventing token overflows well? The implication for us is that we were unintentionally using the new variant of the class including the "refine" prompt, meaning extra tokens and suboptimal configuration (default refine prompt) in our production system.

We are using llama index in a production system and it's hard to detect behavioral changes like this, especially because llama-index has stopped providing detailed release notes since version v0.11.0, at least on https://github.com/run-llama/llama_index/releases?page=2). Are the release notes available somewhere else?

[logan-markewich]

@shadyelgewily-slimstock our changelog is up to date. Updating the github release notes is manual process and sometimes gets missed
https://github.com/run-llama/llama_index/blob/main/CHANGELOG.md

Before, there was no refine, and the chat engine would error out with a token/context window error. Now it doesn't

I recommend pinning versions directly if using in a production system.

[shadyelgewily-slimstock]

Hi @logan-markewich, thanks for your response and I can indeed see the changelog is up to date. I will check there in the future. We also pinned the versions of all llama-index (sub) packages.

I'd like to offer some feedback regarding "CondensePlusContext" as we understand it. For us, this name has a specific meaning: condensing a prompt and adding context from a knowledge base (potentially with self-managed chat history). After reviewing the code, it appears that "CondensePlusContext" now involves both "CondensePlusContext" and "compactAndRefine" approaches, where a prompt is condensed, context is added, and then an iterative compact and refine process may follow. This isn't immediately clear from the name "CondensePlusContext" alone, which raises the question: is it still "CondensePlusContext," or is it more akin to a compactAndRefine approach with additional prompt condensation?

If we had intended to use the compactAndRefine strategy, we would have selected that option. While I fully appreciate that this is your library, and it's still in its early stages, it's worth noting that from a user (and potential contributor) standpoint, changes to strategies with distinct meanings can be confusing. In this case, making the "refine" process optional could still achieve the intended goals without altering the core behavior.

[logan-markewich]

CondensePlusContext means condense the chat history, use it for context retrieval, use the context for responding -- the name is still representative of what it's doing imo?

Most people are surprised when token limits are encountered, because query engines automatically handle token limits, so why shouldn't a chat engine? That was the motivation for the change in any case.

The refine step would only be hit in cases where an error would be raised previously, and with the current context window sizes on most LLMs, would be rarely hit

[shadyelgewily-slimstock]

Based on your response I checked what actually happened as the complete prompt (which does not exceed the token limits) enters the CompactAndRefine instance, and I indeed see that only the _text_qa_template is used when everything fits into a single token limit buffer. The refine prompt is only used when the buffer is exceeded.

Thanks for your answer and we resolved the issue on our end.