[Bugfix] fix output parsing error for trtllm backend

[elinx]

Fix #4136

[ywang96]

Thank you for testing and fix! Do you mind attaching the output from running the fixed version?

[ywang96]

I remember Triton + TRT had an issue where the data['text_output'] contains the cumulative output text instead of the delta, hence initially in this version I only took the last chunk to retrieve generated_text.

Could you verify if this is still the case? (and either way, this should have been data['text_output'] not json.loads(data)["text_output"], so thanks for catching this!)

[elinx]

looks like it has been fixed, the output I got is like:

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":0.0,"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"a"}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"city"}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"in"}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"China"}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0,0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"."}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0,0.0,0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"It"}

data: {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble","model_version":"1","output_log_probs":[0.0,0.0,0.0,0.0,0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"is"}

[elinx]

Thank you for testing and fix! Do you mind attaching the output from running the fixed version?

the final output is something like:

============ Serving Benchmark Result ============
Successful requests:                     1000     
Benchmark duration (s):                  355.37   
Total input tokens:                      248339   
Total generated tokens:                  260518   
Request throughput (req/s):              2.81     
Input token throughput (tok/s):          698.81   
Output token throughput (tok/s):         733.08   
---------------Time to First Token----------------
Mean TTFT (ms):                          213579.15
Median TTFT (ms):                        213038.47
P99 TTFT (ms):                           322949.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.34    
Median TPOT (ms):                        55.59    
P99 TPOT (ms):                           77.75    
==================================================