n_tokens <= n_batch Error

[hafsalm]

Hi,

I'm using the node-llama-cpp package in my project and I've encountered an error that I'm having trouble resolving. The error message is n_tokens <= n_batch.

From what I understand, this error typically occurs when the number of tokens exceeds the batch size. However, I'm not sure how to fix this in my code. Could anyone provide some guidance on what might be causing this error and how to resolve it?

Here's my code(It's a langjs runnable sequence, works well with Ollama, but not working with node-llama-cpp):

import { ChatLlamaCpp } from "langchain/chat_models/llama_cpp";
import { HNSWLib } from "langchain/vectorstores/hnswlib";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import * as fs from "fs";
import { PromptTemplate } from "langchain/prompts";
import { RunnableSequence } from "langchain/schema/runnable";
import { StringOutputParser } from "langchain/schema/output_parser";
import { formatDocumentsAsString } from "langchain/util/document";

/* Initialize the LLM to use to answer the question */
const llamaPath = "mistral-7b-instruct-v0.1.Q4_K_S.gguf"
const model = new ChatLlamaCpp({modelPath: llamaPath})
/* Load in the file we want to do question answering over */
const text = fs.readFileSync("state_of_the_union.txt", "utf8");
/* Split the text into chunks */
const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
const docs = await textSplitter.createDocuments([text]);
/* Create the vectorstore */
const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());
const retriever = vectorStore.asRetriever();

const formatChatHistory = (
  human: string,
  ai: string,
  previousChatHistory?: string
) => {
  const newInteraction = `Human: ${human}\nAI: ${ai}`;
  if (!previousChatHistory) {
    return newInteraction;
  }
  return `${previousChatHistory}\n\n${newInteraction}`;
};

/**
 * Create a prompt template for generating an answer based on context and
 * a question.
 *
 * Chat history will be an empty string if it's the first question.
 *
 * inputVariables: ["chatHistory", "context", "question"]
 */
const questionPrompt = PromptTemplate.fromTemplate(
  `Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
  ----------------
  CONTEXT: {context}
  ----------------
  CHAT HISTORY: {chatHistory}
  ----------------
  QUESTION: {question}
  ----------------
  Helpful Answer:`
);

const chain = RunnableSequence.from([
  {
    question: (input: { question: string; chatHistory?: string }) =>
      input.question,
    chatHistory: (input: { question: string; chatHistory?: string }) =>
      input.chatHistory ?? "",
    context: async (input: { question: string; chatHistory?: string }) => {
      const relevantDocs = await retriever.getRelevantDocuments(input.question);
      const serialized = formatDocumentsAsString(relevantDocs);
      return serialized;
    },
  },
  questionPrompt,
  model,
  new StringOutputParser(),
]);

const questionOne = "What did the president say about Justice Breyer?";

const resultOne = await chain.invoke({
  question: questionOne,
});

console.log({ resultOne });
/**
 * {
 *   resultOne: 'The president thanked Justice Breyer for his service and described him as an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court.'
 * }
 */

const resultTwo = await chain.invoke({
  chatHistory: formatChatHistory(resultOne, questionOne),
  question: "Was it nice?",
});

console.log({ resultTwo });
/**
 * {
 *   resultTwo: "Yes, the president's description of Justice Breyer was positive."
 * }
 */ 

[OlleMattsson]

The LlamaContextOptions allows you to set the batch size.

https://withcatai.github.io/node-llama-cpp/api/type-aliases/LlamaContextOptions#type-alias-llamacontextoptions

eg.

const context = new LlamaContext({
      batchSize: 1024,
 });

Hope that helps. I unable to give you guidance as to how exactly implement that in your particular code.

[hafsalm]

@OlleMattsson Thank you so much, that worked.

const chatllmModel = new ChatLlamaCpp({
  modelPath: llamaPath,
  batchSize:1024
})

Here is how I used the batchSize in my code. Just leaving it here in case someone else is on the same hunt.