how to use openbmb/MiniCPM3-4B-GGUF

[gy9527]

I am trying to use openbmb/MiniCPM3-4B-GGUF, but the model doesn't stop generating output. After checking the documentation, I learned that I need to customize the chat wrapper. But how should I define the format? Can you provide some help?

[giladgd]

I just tested this model and it worked as expected.
You can run this command to test it yourself:

npx -y node-llama-cpp chat --prompt 'Hi there!' https://huggingface.co/openbmb/MiniCPM3-4B-GGUF/blob/main/minicpm3-4b-q4_k_m.gguf

If you could share the code you've used and the result of running the npx -y node-llama-cpp inspect gpu command then I can try to help you find what the issue is.

I recommend you to try to scaffold a new project and run it with this model.

[gy9527]

I previously tested the 'qwen2.5' series, and everything was OK, but this model cannot stop generating output.

import { fileURLToPath } from "node:url"
import * as path from "node:path";
import { getLlama, LlamaChatSession } from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const modelPath = path.join(__dirname, "./models/", "minicpm3-4b-q4_k_m.gguf")

const llama = await getLlama();
const model = await llama.loadModel({
    modelPath
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const q1 = "Hi there, how are you?";

const a1 = await session.prompt(q1);

console.log(a1)

---

OS: Windows 10.0.22631 (x64)
Node: 20.16.0 (x64)
node-llama-cpp: 3.1.1

Vulkan: available

Vulkan device: Intel(R) Arc(TM) Graphics
Vulkan used VRAM: 0% (0B/15.75GB)
Vulkan free VRAM: 100% (15.75GB/15.75GB)

CPU model: Intel(R) Core(TM) Ultra 9 185H
Math cores: 16
Used RAM: 34.64% (10.92GB/31.51GB)
Free RAM: 65.35% (20.59GB/31.51GB)