Langchain GPU

[PeterTucker]

Hi, I already asked this question on the Langchain Slack chat, and the developers were not sure. Is there any way to load in a model using langchain and tell node-llama-cpp to use the GPU? I also saw this:

There is an option for node-llama-cpp to run a download to install a metal or cuda version of llama.cpp, npx node-llama-cpp download —metal. read me. I can always add a mention to the docs.
langchain-ai/langchainjs#2413 (comment)

[giladgd]

Currently, Langchain only supports node-llama-cpp version 2, not version 3.
Version 3 introduced seamless GPU integration that doesn't require any configuration or manual actions to use all the available hardware, including GPUs.

On version 2, Metal is used by default on Apple Silicone devices, so there's no need to build anything to use it, but for all other OSs and architectures you have to manually build from source using the command you mentioned with relevant flags.
Version 2 is not supported anymore and won't receive any updates, thus it also won't be compatible with newer llama.cpp versions so new models cannot be used with it.
I plan to update the node-llama-cpp integration of Langchain to support version 3, but it may take some time since I don't have enough free time for it at the moment, and I'm currently occupied with other tasks on node-llama-cpp I have to finish before I can attend to this.
Maybe the Langchain maintainers would want to update the integration themselves in the meantime.
You can also do it if you want :)