Why vllm-backend prepends output with the prompt?

[mkhludnev]

Here's the concatenation
Here's how output looks like for REST and gRPC
Bug on server

[oandreeva-nv]

I agree that we most likely need to remove prompt concatenation and just send the generated response. Alternatively, we can introduce keep prompt flag in config and default it to false. CC @nnshah1 @rmccorm4 @GuanLuo @Tabrizian @krishung5

[oandreeva-nv]

@mkhludnev trt-llm backend has exclude_input_in_output flag, that controls prompt concatenation. We will add same flag to vLLM soon

[mkhludnev]

Wow! Great. May I kick tiers under #6866 and come up with PR?

[oandreeva-nv]

@mkhludnev I am working on this PR for vLLM backend, it will be up by the end of this week. I'll tag you for review, if you are interested.
May I kindly ask you to be consistent with opening and closing issues, please. General suggestion is to keep them open until the issue is resolved. This helps with the internal task management/prioritization as well.

[mkhludnev]

for the reference triton-inference-server/vllm_backend#35