Why prepending output with prompt?

[mkhludnev]

I'd rather created issue for discussion, but this repo doesn't have issues enabled. First of all such prepending seems redundant especially for long RAG prompts. Then, it's an actual problem since I notice that triton gRPC crops the long response. Curiously, REST doesn't crop payload and full concatenation of prompt and output arrives to client.

Here I put more details of the issue langchain-ai/langchain#12474 (comment)

[mkhludnev]

@oandreeva-nv hello, may I ask you a question? 👆

[oandreeva-nv]

Hi @mkhludnev, our ReadMe mentions that you can report all issues on issues page, which is for server. Apologies for confusion, we use server repo as the main entry-point for questions & issues.

Regarding your question, I believe the main goal is to report the full text prompt + generated portion. Do you happen to have a reproducer for gRPC cropping the part of the output? If it does, I am concerned that simply removing prompt from the response may not work

[mkhludnev]

Thanks for replying @oandreeva-nv

report all issues on issues page, which is for server.

Here we go triton-inference-server/server#6866 Perhaps it's worth to discuss first.

Regarding your question, I believe the main goal is to report the full text prompt + generated portion. Do you happen to have a reproducer for gRPC cropping the part of the output?

Well. I just have triton container, curl, and python app, which logs gRPC output. I put output excerpts into the linked langchain issue above.

If it does, I am concerned that simply removing prompt from the response may not work

Right.

[mkhludnev]

Considering triton-inference-server/server#6867 it's worth to implement echo=False option for vlm and trt backends.