[Misc]: output text is not match with first top_logprobs item on stream mode.

[keyword1983]

Anything you want to discuss about vllm.

Env:
vllm: v0.4.1-v0.5.0

request parms
{
"prompt": (
"function_metadata\n\nAssistant have access to the following functions. Use them if required:\n"
"[{"type": "function", "function": {"name": "calculate_triangle_area", "description": "Calculate the area of a triangle given its base and height.", "parameters": {"type": "object", "properties": {"base": {"type": "number", "description": "The base of the triangle."}, "height": {"type": "number", "description": "The height of the triangle."}, "unit": {"type": "string", "description": "The unit of measure (defaults to 'units' if not specified)"}}, "required": ["base", "height"]}}}]user\n\n"
"Find the area of a triangle with a base of 10 units and height of 5 units.<|start_header_id|>assistant\n\n"
),
"max_tokens": 500,
"temperature": 0.5,
"repetition_penalty": 1.0,
"presence_penalty": 0.0,
"top_k": 50,
"logprobs": False,
"stop":['<|eot_id|>'],
"skip_special_tokens":True,
"spaces_between_special_tokens":True,
"stream":True,
}

Streaming response:
data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"<|too","logprobs":{"text_offset":[0],"token_logprobs":[0.0],"tokens":["<|tool_calls|>"],"top_logprobs":[{"<|tool_calls|>":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"l","logprobs":{"text_offset":[5],"token_logprobs":[0.0],"tokens":["["],"top_logprobs":[{"[":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"_c","logprobs":{"text_offset":[6],"token_logprobs":[0.0],"tokens":["{""],"top_logprobs":[{"{"":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"al","logprobs":{"text_offset":[8],"token_logprobs":[0.0],"tokens":["id"],"top_logprobs":[{"id":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"ls","logprobs":{"text_offset":[10],"token_logprobs":[0.0],"tokens":["":"],"top_logprobs":[{"":":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-02313246eab24d57b890ad13fd0b57f0","created":1718849637,"model":"/models/llama3-8b-instruct-ft-0522/","choices":[{"index":0,"text":"|>","logprobs":{"text_offset":[12],"token_logprobs":[0.0],"tokens":[" ""],"top_logprobs":[{" "":0.0}]},"finish_reason":null,"stop_reason":null}],"usage":null}

In stream mode, we observe that that <|tool_calls|> divided into multiple response outputs. and the content of the response output text does not match the content in top_logprobs. This issue was caused by commit #3672 . This discrepancy can cause problems for client users.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!