server: fix the disappearance of the end of the text when streaming with stop strings #9867
+4
−5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The problem: the end of the text may not be pushed to the API-client when streaming with stop strings enabled.
For example, let's test the following prompt:
The stop string will be
\n
(stopping at the end of a dialog line). The next predicted character should be the question mark (it actually depends on the model and the probabilities, but in this case, let's assume that theseed
is chosen so that the next character is?
).The bug depends heavily on the model's tokenizer. In my case the bug can be seen on
Llama-3.2-3B-Instruct-Q8_0.gguf
model.First, let's try it without streaming:
It gives the correct result:
Now let's try it with streaming:
The question mark disappeared. And it's also not featured in the
stopping_word
. So it's completely lost and the API-client won't be able to restore it.It happens because the next token returned by the model contains both a question mark and the stop string:
?\n\n
. And the current code skips sending the current token completely if it contains the stop string.The change in this PR sends the remainder of the token to the API-client in this case. When
is_stop_full=true
it's safe to send the response, because the stop string and everything after it will already be truncated from thegenerated_text
at this point.The response with this PR applied: