Very good experience to use LangChain together with TensorRT-LLM backend

[npuichigo]

I build a service (https://github.com/npuichigo/openai_trtllm) to provide an OpenAI compatible endpoint for TensorRT-LLM triton backend, to be used together with LangChain or other popular community LLM tools, and it shows really good performance compared to FastChat. Thanks for the great work of TensorRT-LLM.

[juney-nvidia]

@npuichigo

Hi, it is glad to hear that TensorRT-LLM can help you achieve great performance and provide value for your use case.

There are more and more new features, enhancements and optimizations coming on the way, hoping you will like it:)

And welcome for providing any feedbacks to the TensorRT-LLM team.

Thanks
June

[jdemouth-nvidia]

@npuichigo thanks a lot for the great feedback! Do not hesitate if June (@juney-nvidia) or myself can help you in any way. We want everyone to have a good experience with TensorRT-LLM.

[nnshah1]

@npuichigo awesome to see this integration. Is this something that could be added to the langchain repository?

[npuichigo]

@nnshah1 Yes, it provides TensorRT-LLM an OpenAI compatible API (as a gateway), and is a replacement of FastChat. So it's compatible with openai sdk (https://github.com/npuichigo/openai_trtllm/blob/main/client/openai_completion_stream.py). You can use ChatOpenAI or OpenAI in langchain which actually calls the openai sdk, so the request will be sent to triton backend through this gateway.

[npuichigo]

Currently I tested with main branch and it works well, but the work is not 100% done. I'm planning to provide a runnable example for people to play with.