Triton LangChain Provider

[peterableda]

Is your feature request related to a problem? Please describe.
LangChain is one of the most widely used libraries to build LLM based applications with a wide range of integrations to LLM providers. Triton Inference Server should be supported within that community.

I would like to host open source LLMs from HuggingFace in Triton as a Coding Assistant for JupyterLab. I would use the Jupyter-AI extension that already integrates with a wide range of LLM Providers, it relies on LangChain to abstract the interactions.

Describe the solution you'd like
Build an NVIDIA Triton Inference Server LangChain Provider.

Additional context
A similar request was raised in the LangChain community: langchain-ai/langchain#12474

I also saw LangChain coming up as an alternative solution in the #6583 request.

[fpetrini15]

Thank you @peterableda for submitting a feature request! I have filed a ticket for us to discuss and investigate further.

[nnshah1]

@peterableda,

@npuichigo has posted this on the TRT-LLM discussion.

NVIDIA/TensorRT-LLM#591

[npuichigo]

@peterableda,

@npuichigo has posted this on the TRT-LLM discussion.

NVIDIA/TensorRT-LLM#591

Welcome to have a try with https://github.com/npuichigo/openai_trtllm if anyone want an openai-like API to use with LangChain.

[peterableda]

This seems to be the same/v similar request: langchain-ai/langchain#13975

[dyastremsky]

Closing issue due to inactivity. Please reopen if you would like to follow up with this issue.