NoBinaryFoundError for Windows in Electron when Upgrading from 3.0.0-beta44 -> 3.2.0

[bitterspeed]

Issue description

NoBinaryFoundError for Windows when Upgrading from 3.0.0-beta44 -> 3.2.0

Expected Behavior

I'd expect the WIndows x64 prebuilt binary to be available for use so users do not have to build locally.

Actual Behavior

A NoBinaryFoundError is thrown when trying to start getLlama

Steps to reproduce

• Use Electron + node-llama-cpp
• build on windows
• use getLlama()

Btw, prebuilt binary was found and worked great in 3.0.0-beta44

My Environment

Dependency	Version
Operating System	Win 10
CPU	Intel I7
node-llama-cpp version	3.2.0

Additional Context

No response

Relevant Features Used

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

[giladgd]

@bitterspeed Can you please attach the console logs from the main process of your app?
I don't have enough information to investigate this, and it doesn't reproduce on my Windows machine.
Also, please attach the output from running this command:

npx --yes node-llama-cpp inspect gpu

[bitterspeed]

Running the app unbundled works fine with 3.2.0. When I make it using electron-forge and ship node-llama-cpp as a separate module (unpacked from asar), I get the NoBinaryFoundError below.

The below works great in 3.0.0-beta44 for Mac + Win, but fails for Win (not Mac) in 3.2.0. This is after making.

const winConfig: ForgeConfig = {
  packagerConfig: {
    asar: {
      unpack: '**/node_modules/node-llama-cpp/**',
    },
    icon: './src/assets/Icon',
    protocols: [
      {
        name: 'socrates',
        schemes: ['socrates'],
      },
    ],
  },
  rebuildConfig: {},
  makers: [
    new MakerSquirrel(
      {
        setupExe: 'Socrates.exe',
        setupIcon: path.join(__dirname, '/src/assets/Icon.ico'),
      },
      ['win32']
    ),
    new MakerRpm({}),
    new MakerDeb({}),
  ],
  publishers: [
    {
      name: '@electron-forge/publisher-github',
      config: {
        repository: {
          owner: process.env.GITHUB_OWNER,
          name: process.env.GITHUB_REPO,
        },
        prerelease: false,
        draft: true,
      },
    },
  ],
  plugins: [
    new AutoUnpackNativesPlugin({}),
    new VitePlugin({
      // `build` can specify multiple entry builds, which can be Main process, Preload scripts, Worker process, etc.
      // If you are familiar with Vite configuration, it will look really familiar.
      build: [
        {
          // `entry` is just an alias for `build.lib.entry` in the corresponding file of `config`.
          entry: 'src/main.ts',
          config: 'vite.main.config.ts',
        },
        {
          entry: 'src/preload.ts',
          config: 'vite.preload.config.ts',
        },
      ],
      renderer: [
        {
          name: 'main_window',
          config: 'vite.renderer.config.ts',
        },
      ],
    }),
    new ElectronForgeAzureSignToolPlugin({
      azureKeyVaultUri: process.env.AZURE_KEY_VAULT_URL || '',
      azureClientId: process.env.AZURE_CLIENT_ID || '',
      azureTenantId: process.env.AZURE_TENANT_ID || '',
      azureClientSecret: process.env.AZURE_CLIENT_SECRET || '',
      azureCertificateName: process.env.AZURE_CERT_NAME || '',
    }),
    // Fuses are used to enable/disable various Electron functionality
    // at package time, before code signing the application
    new FusesPlugin({
      version: FuseVersion.V1,
      [FuseV1Options.RunAsNode]: false,
      [FuseV1Options.EnableCookieEncryption]: true,
      [FuseV1Options.EnableNodeOptionsEnvironmentVariable]: false,
      [FuseV1Options.EnableNodeCliInspectArguments]: false,
      [FuseV1Options.EnableEmbeddedAsarIntegrityValidation]: true,
      [FuseV1Options.OnlyLoadAppFromAsar]: true,
    }),
  ],
};

npx --yes node-llama-cpp inspect gpu

OS: Windows 10.0.19045 (x64)
Node: 18.20.1 (x64)
node-llama-cpp: 3.2.0

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes
CUDA: available
Vulkan: available

CUDA device: NVIDIA GeForce RTX 3080
CUDA used VRAM: 11.39% (1.14GB/10GB)
CUDA free VRAM: 88.6% (8.86GB/10GB)

Vulkan devices: NVIDIA GeForce RTX 3080, NVIDIA GeForce RTX 3080
Vulkan used VRAM: 2.07% (212.94MB/10.03GB)
Vulkan free VRAM: 97.92% (9.82GB/10.03GB)

CPU model: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
Math cores: 8
Used RAM: 35.25% (11.24GB/31.89GB)
Free RAM: 64.74% (20.65GB/31.89GB)
Used swap: 38.87% (14.25GB/36.64GB)
Max swap size: 36.64GB

Electron app logs when calling getLlama inside main process

  const { getLlama } = await import('node-llama-cpp');
  try {
    const llama = await getLlama({
      build: 'auto',
    });

[2024-11-05 10:06:27.238] [error] Error: NoBinaryFoundError
    at getLlamaForOptions (file:///C:/Users/bitterspeed/AppData/Local/electron/app-2.3.10/resources/app.asar/node_modules/node-llama-cpp/dist/bindings/getLlama.js:170:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Yi (C:\Users\bitterspeed\AppData\Local\electron\app-2.3.10\resources\app.asar\.vite\build\main-DP2mZX9g.js:59:1888)
    at async WebContents.<anonymous> (node:electron/js2c/browser_init:2:77963)

[giladgd]

You haven't experienced this issue before since in 3.0.0-beta.44 most of the binaries were shipped as part of the main node-llama-cpp module.
Since the build size of the native binaries grew, I had to extract them to separate modules under the @node-llama-cpp scope:

node-llama-cpp/package.json

Lines 214 to 226 in 6405ee9

	"optionalDependencies": {
	"@node-llama-cpp/linux-arm64": "0.1.0",
	"@node-llama-cpp/linux-armv7l": "0.1.0",
	"@node-llama-cpp/linux-x64": "0.1.0",
	"@node-llama-cpp/linux-x64-cuda": "0.1.0",
	"@node-llama-cpp/linux-x64-vulkan": "0.1.0",
	"@node-llama-cpp/mac-arm64-metal": "0.1.0",
	"@node-llama-cpp/mac-x64": "0.1.0",
	"@node-llama-cpp/win-arm64": "0.1.0",
	"@node-llama-cpp/win-x64": "0.1.0",
	"@node-llama-cpp/win-x64-cuda": "0.1.0",
	"@node-llama-cpp/win-x64-vulkan": "0.1.0"
	}

The reason you get this error seems to be that the prebuilt binaries under the @node-llama-cpp scope are not packaged as unpacked in the final app.
In the Electron template I've used electron-builder, but I recommend you to take a look at its config to see how to configure the packaging of the prebuilt binaries:

node-llama-cpp/templates/electron-typescript-react/electron-builder.ts

Lines 33 to 47 in 6405ee9

	files: [
	"dist",
	"dist-electron",
	"!node_modules/node-llama-cpp/bins/**/*",
	"node_modules/node-llama-cpp/bins/${os}-${arch}*/**/*",
	"!node_modules/@node-llama-cpp/*/bins/**/*",
	"node_modules/@node-llama-cpp/${os}-${arch}*/bins/**/*",
	"!node_modules/node-llama-cpp/llama/localBuilds/**/*",
	"node_modules/node-llama-cpp/llama/localBuilds/${os}-${arch}*/**/*"
	],
	asarUnpack: [
	"node_modules/node-llama-cpp/bins",
	"node_modules/node-llama-cpp/llama/localBuilds",
	"node_modules/@node-llama-cpp/*"
	],

I also recommend you to take a look at the Electron documentation of node-llama-cpp, and in particular on the cross compilation section, since it has changed since the beta version and might affect your build process.

[bitterspeed]

Thanks, this is helpful, but shouldn't the optional dependencies be inside the /node_modules/node-llama-cpp/node_modules folder after npm i node-llama-cpp?

Note that when running npm i node-llama-cpp on mac, a mac-arm64-metal is included in the node-llama-cpp/llama/localBuilds folder AND node_modules/node-llama-cpp/node_modules/@node-llama-cpp, but in Win, there is neither of these.

In Win, I had to explicitly install those dependencies for it to be bundled inside node_modules/node-llama-cpp/node_modules/@node-llama-cpp. (App bundle now works, thanks)

EDIT: I had a nohoist setting in package.json, which was causing the issue. All good now, thanks!

[giladgd]

npm attempts to flatten the modules in the node_modules directory as much as possible by default.
When a module is required to be installed at 2 conflicting versions by other modules, npm will install one of the conflicting versions locally inside a nested node_modules under one of the modules that need it.
This ensures the node_modules directory is as concise as possible by reusing the same module installation by multiple other modules that depend on it.

When installing node-llama-cpp, it tests the prebuilt binary to ensure it's compatible with your machine, and in case it fails the test it then builds from source.
If you always see a build under the node_modules/node-llama-cpp/llama/localBuilds directory after installing node-llama-cpp on an empty node_modules directory, then your machine may not be compatible with the prebuilt binaries for some reason (and I'd like to find out why so I can fix it).
If you use a package manager other than npm it may also be the reason for this, since some package managers use symlinks to a global installation of modules, which can cause the changes made to node-llama-cpp in one project to affect another projects (which is why I only provide examples using npm in the documentation).

To check whether your machine is compatible with the prebuilt binaries, run these commands:

npx --no node-llama-cpp source clear
npx --no node-llama-cpp chat --prompt 'Hi there!'

And then select some model.

If it attempts to build from source, you'll see logs for the build progress, which means the prebuilt binaries failed the test and aren't compatible with your machine.
If the model loaded successfully without building from source then the prebuilt binaries are compatible with your machine and were used for the chat session.

You shouldn't manually install anything under the @node-llama-cpp scope, as these should be installed automatically by npm, so only the packages relevant to the current OS are installed, and their versions will always be in sync with node-llama-cpp.
Maybe your npm version is too old to do that?

[bitterspeed]

I appreciate the explanation. I am using NPM 10.2.4.

A cleaner solution to the above that worked for me: rm -rf node_modules, and upon reinstallation, it properly installed both node_modules/node-llama-cpp and node_modules/@node-llama-cpp/mac-arm64-metal

Since i'm using Electron Forge I had to manually copy over the node_modules/@node-llama-cpp optional dependency for the Electron app to work (no more NoBinaryFoundError!).

Inside forge.config.ts

    afterCopy: [
      (buildPath, electronVersion, platform, arch, callback) => {
        const sourcePath = path.join(
          process.cwd(),
          'node_modules/@node-llama-cpp'
        );
        const targetPath = path.join(buildPath, 'node_modules/@node-llama-cpp');

        try {
          ensureDirSync(targetPath);
          copySync(sourcePath, targetPath);
          console.log(`Copied node-llama-cpp dependencies to ${targetPath}`);
        } catch (error) {
          console.error('Error copying node-llama-cpp dependencies:', error);
        }

        callback();
      },

With that said, I cannot get CUDA to work on Windows even if it's properly bundled in ASAR unpacked.

    const llama = await getLlama();
    log.info("GPU type:", llama.gpu);

Shows as GPU type: metal on Mac, but GPU type: false on Win if I include @node-llama-cpp/win-x64, @node-llama-cpp/win-x64-vulkan, @node-llama-cpp/win-x64-cuda in ASAR unpacked.

That is, I seem to be able to use the win-x64 unpacked pkg but not win-x64-cuda in Electron prod.

I am able to use win-x64-cuda when running Electron in Dev.

[giladgd]

Try to inspect the resource directory of the prod Electron app to ensure win-x64-cuda is located under app.asar.unpacked/node_modules/@node-llama-cpp and that the bin directory inside it is identical to the bin directory from the original module (including all file names and sizes).
Also, try to launch the app from the terminal to see logs from the main process, to check whether there is some issue with the packaged binary that's logged there.

You can also download a build of the example electron app template from the latest release of node-llama-cpp and inspect the files of it (use the binary format and inspect the files it unpacks after installation) to see what's different in your build, and in particular the app.asar file.
I recommend unpacking it to inspect its content using this command (run it inside the resource directory):

npx asar extract app.asar app.content

[bitterspeed]

Thanks for the tips.

My app.asar.unpacked/node_modules/@node-llama-cpp and the bin directory inside is identical to all files in the original module

My App's app.asar.unpacked

I downloaded the electron template app, installed node-llama-cpp-electron-example.Windows.3.2.0.x64.exe.

First, I compared the template app's app.asar.unpacked

to my app's app.asar.unpacked and they are identical

I ran npx asar extract app.asar app.content on the template app, but it seems like the resulting app.content includes files from both app.asar.unpacked and app.asar? Is it supposed to, or did I do something incorrectly?

The original app.asar size is only 42MB, so seems like it takes from app.asar.unpacked

Electron Forge's Squirrel.Windows maker doesn't seem to 'install' it the same way the template does in the sense that I can just double click on my app's .exe to open it, while the template's exe looks to be a more traditional Windows installer. Could that be the issue?

[giladgd]

I think the issue you're facing is related to how the asar is packed in your build.
The asar file should be packed with the final node_modules directory with some directories marked as unpacked as part of the packing process in order for those directories to be included in the asar header.
File that are put in the unpacked directory externally (after the packing process) are not recognized since they do not appear in the asar header.
Hence I reckon it's better to ensure the all the required files and folders are packed properly in the first place than to add them manually in the build process.

Try to run the asar extract command on the asar of your app after installing it to see whether the win-x64-cuda directory is there.
If it's not there then it's not being recognized by your app.

Also, since you mentioned that it works in dev mode but not in a packaged app, it's possible that the CUDA version installed on your machine is too old or incompatible with the prebuilt CUDA binaries, so it builds from source when running in dev mode (since you have build: "auto" in your getLlama options).
If the CUDA binaries from the local build are not included in the final asar package then it would have no compatible CUDA binaries available to be used, so it fallbacks to binaries with no CUDA support.
You can see in the documentation that it'll never build from source when running from an asar archive (even with build: "auto"), which is due to the asar archive being read-only.

[bitterspeed]

Even though I manually copy the @node-llama-cpp packages, this is technically during the packaging step (L 159 is where all the other node_modules are copied. afterCopy, the hook I use in the config file, runs at L 163) and before ASAR packaging (before it hits electron/asar, L239)

I need to use afterCopy b/c @node-llama-cpp is an optional dependency and is not included by default with vite as a result.

By putting in the afterCopy code above, which runs at line 163, it reads the win-x64 package but does not read win-x64-cuda. I do NOT get a NoBinaryFoundError using afterCopy, but without afterCopy, the NoBinaryFoundError understandably shows up.

This is the strange part for me why it works withwin-x64 but not win-x64-cuda.

Try to run the asar extract command on the asar of your app after installing it to see whether the win-x64-cuda directory is there.
If it's not there then it's not being recognized by your app.

Yes, running asar extract with the command above outputting to app.content on my app does show the win-x64-cuda directory in there.

My CUDA version:

Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Apr_17_19:36:51_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.5, V12.5.40
Build cuda_12.5.r12.5/compiler.34177558_0

There are no local build files inside in dev directory: apps/electron/node_modules/node-llama-cpp/llama, so I don't believe it is reading from the local build but reading from @node-llama-cpp/win-x64-cuda.

[giladgd]

I need to use afterCopy b/c @node-llama-cpp is an optional dependency and is not included by default with vite as a result.

I think the reason the @node-llama-cpp modules are not included by default has something to do with the build configuration.
These modules being marked as optional dependencies is relevant only for the installation process from npm; those modules are dependencies that should be treated like any other module dependencies for any other purpose.
Using optional dependencies for prebuilt binaries is a common pattern used by many native modules, so I don't think the optional dependencies need special care in the build process.

I think you may not have marked node-llama-cpp as an external module, as suggested in the Electron Forage Vite documentation, which causes Vite to attempt to bundle it.
Try to inspect the final build to see how node-llama-cpp is used.
It should be imported directly using import {...} from "node-llama-cpp" or import("node-llama-cpp").

To debug, try to import the modules of the prebuilt binaries directly and see whether the import is successful in the final build:

try {
    const importRes = await import("@node-llama-cpp/win-x64-cuda");
    const {binsDir, packageVersion} = importRes.getBinsDir();
    console.log("CUDA binaries module binsDir", binsDir);
    console.log("CUDA binaries module version", packageVersion);
    // if the version doesn't match the version of `node-llama-cpp`, then this module won't be used
} catch(err) {
    console.log("CUDA binaries module is not found", err);
}

This code should help us understand whether the prebuilt binaries are accessible to be imported in the final build.

This is the strange part for me why it works withwin-x64 but not win-x64-cuda

This indeed seems strange; I'd love to find out why it happens so I can fix it, or document better how to fix it.
When you used the example Electron template app build from the latest release, did it use CUDA on your machine?
You can use nvidia-smi to see the load on the GPU to check that.

Also, try using getLlama({gpu: "cuda"}) to force it only to use the CUDA binaries (maybe it doesn't detect CUDA availability properly for some reason) - let me know whether it was successful.

[bitterspeed]

Including this code

try {
    const importRes = await import("@node-llama-cpp/win-x64-cuda");
    const {binsDir, packageVersion} = importRes.getBinsDir();
    console.log("CUDA binaries module binsDir", binsDir);
    console.log("CUDA binaries module version", packageVersion);
    // if the version doesn't match the version of `node-llama-cpp`, then this module won't be used
} catch(err) {
    console.log("CUDA binaries module is not found", err);
}

causes this error when running npm run make:

An unhandled rejection has occurred inside Forge:
Error: The main entry point to your app was not found. Make sure "C:\g\socrates\apps\electron.vite\build\main.js" exists and does not get ignored by your ignore option
at validateElectronApp (C:\g\socrates\apps\electron\node_modules@electron\packager\src\common.ts:102:11)
at async WindowsApp.buildApp (C:\g\socrates\apps\electron\node_modules@electron\packager\src\platform.ts:149:5)
at async WindowsApp.initialize (C:\g\socrates\apps\electron\node_modules@electron\packager\src\platform.ts:141:7)
at async WindowsApp.create (C:\g\socrates\apps\electron\node_modules@electron\packager\src\win32.ts:92:5)
at async Promise.all (index 0)
at async packager (C:\g\socrates\apps\electron\node_modules@electron\packager\src\packager.ts:246:20)

node-llama-cpp has already been marked as external, which is in my package.json dependencies:

Also, even if I explicitly include '@node-llama-cpp/win-x64-cuda', it's still not being used despite it being copied over.
"node-llama-cpp": "^3.2.0",

export const external = [
  ...builtins,
  ...Object.keys(
    'dependencies' in pkg ? (pkg.dependencies as Record<string, unknown>) : {}
  ),
];

Yes, running the template uses CUDA:

Mon Nov 11 10:23:08 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.94 Driver Version: 560.94 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3080 WDDM | 00000000:01:00.0 On | N/A |
| 66% 61C P2 315W / 370W | 4905MiB / 10240MiB | 78% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

---

Also, try using getLlama({gpu: "cuda"}) to force it only to use the CUDA binaries (maybe it doesn't detect CUDA availability properly for some reason) - let me know whether it was successful.

const llama = await getLlama({gpu: 'cuda'}); shows the NoBinaryFound error as expected.

---

I think the reason the @node-llama-cpp modules are not included by default has something to do with the build configuration.
These modules being marked as optional dependencies is relevant only for the installation process from npm; those modules are dependencies that should be treated like any other module dependencies for any other purpose.
Using optional dependencies for prebuilt binaries is a common pattern used by many native modules, so I don't think the optional dependencies need special care in the build process.

It may indeed have something to do with the way it's imported & the fact that it's optional (?).

When I look at what is copied over to the final bundle before ASAR packaging, optional dependencies of other packages are included, but not node-llama-cpp's (ie, @node-llama-cpp/*). That's why I have to use afterCopy to avoid the NoBinaryFoundError.

Currently I'm using const { getLlama } = await import('node-llama-cpp'); Importing node-llama-cpp directly import { getLlama} from 'node-llama-cpp' prevents my electron app from running.

[giladgd]

Can you please create a minimal reproducible example repo that I can use to poke around?
It'll make it much easier for me to find the issue and its solution this way.

I suspect that the code of node-llama-cpp is still being bundled/transpiled in some way in the final build, despite being marked as an external module, which may cause the original module from the node_modules directory to not be used despite being included in the asar.

The code snippet you included in your build is not supposed to fail the build, so since it did, then it means there's some transpilation that happens on the import function, which I think may be the cause for this.

[bitterspeed]

Greatly appreciate the help. Here's the repo:

In processRag.ts, change TEST_PATH to where a chat GGUF model is stored (e.g., Llama-3.2-3B-Instruct-Q4_K_M.gguf):

      const TEST_PATH =
        process.platform === 'darwin'
          ? '/Users/goodspeed/Library/Application Support/Socrates'
          : 'C:\\Users\\goodspeed\\AppData\\Roaming\\Socrates';
      llamaModel = await llama.loadModel({
        // modelPath: join(app.getPath('sessionData'), `models`, queryInput.model),
        modelPath: join(TEST_PATH, `models`, queryInput.model),
      });

To create this repo, I ran:

1. npm init electron-app@latest desktop -- --template=vite-typescript
2. Added node-llama-cpp and electron-log
3. Edited the vite and forge config files to unbundle node-llama-cpp
4. Added basic IPC call to query model

Using this repo, CUDA works great in Windows development. Mac works great in Prod + Development. In Windows prod, it says it's missing some dependencies (from node-llama-cpp):

Uncaught (in promise) Error: Error invoking remote method 'processRag': Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'lifecycle-utils' imported from C:\Users\goodspeed\AppData\Local\desktop\app-2.4.1\resources\app.asar\node_modules\node-llama-cpp\dist\index.js

Perhaps if I individually copied each of node-llama-cpp's dependencies in afterCopy in forge.config.ts, it'll work? I doubt this is the right approach though.

Note this error message is different behavior from what I posted above, where node-llama-cpp works, but the CUDA binary is not used. With that said, my code above is using an older version of Electron and electron-forge, but I'm happy to rebuild my project using the newer version of Electron as the basis.