Give Leo information about when text is being truncated so it can better formulate an answer.

[bbondy]

Could we do better at making the model aware that it didn’t consume the full content? I didn’t verify but I’m assuming it stopped at 13 because the text got truncated?
https://music.youtube.com/playlist?list=OLAK5uy_lm33KX5L-19O27gjy9IrZCQJEFxx00WMQ

[petemill]

Is it enough to make the user aware it's cut off? #31405

[bbondy]

For MVP? I think so

[stevelaskaridis]

After some testing, here are my findings:

List of links tested

• https://www.theguardian.com/world/live/2023/nov/28/israel-hamas-war-live-updates-hostages-palestinians-released-gaza-ceasefire-extended
• https://www.quantamagazine.org/meet-strange-metals-where-electricity-may-flow-without-electrons-20231127/
• https://www.downtowndougbrown.com/2023/11/how-apples-developers-reflashed-mac-roms-in-the-90s/
• https://www.campsites.co.uk/downloads/the-ultimate-camping-checklist
• https://www.youtube.com/watch?v=JN3KPFbWCy8

Methodology

Tested summary plus a few questions about the context inside and outside of scope.

Prompt changes tested:

(changes signified in bold)

• Mode 1 (prepend): This is part of an article/transcript within <article> tags: [...]
• Mode 2 (apppend): [...] You have only seen part of the input.
• Mode 3 (seed): [/INST] You have only read part of this input. Here is your response:

Models checked

• Llama-2-13B-chat
• Llama-2-70B-chat

Results

Mode 1 - Prepend

Works okay, does not make much of a difference, unless you explicitly ask if the whole content was consumed. 13B model does not answer that correctly (assumes the whole article was read).

Mode 2 - Append

Works okay. Stronger conditioning that the whole output was not consumed.
Sometimes, it makes the model not to respond properly.

Mode 3 - Seed

This did not work at all. It affects the task completion significantly.

Recommendation

Making the prompt truncation aware does not monotonically change the behaviour towards better responses. However, under some queries, it may yield that the whole content was not consumed instead of e.g. the cutoff point of training or lack of access to real information. The downside is that I also saw the behaviour of refusing to respond due to "ethics, etc.".

For Claude-Instant, I did not test thoroughly as the context limit is quite high anyways.

If we are to integrated it, I would vote for Mode 2. I can push the changes made. QA can further test more wide into the impact of this prompt change.

[bbondy]

An option is to just close this with that investigation too. Do you recommend that or would you prefer Mode 2?

[bbondy]

Keeping in mind that we do warn the user about it now even if the model doesn't know.

[stevelaskaridis]

Given that Llama-13B does not particularly "care" about this prompt change anyways (which is our public offering), I propose that we close this for now.

[bbondy]

thanks for checking 👍