add support for batched inference

[promptboom]

as llama.cpp supports batched inference in 8.26, can llama-cpp-python supports for batched inference? I just tried the latest version 0.2.11 and found that it is not possible to make multiple requests simultaneously.

[yourbuddyconner]

Linking the PRs for visibility:

• llama : custom attention mask + parallel decoding + no context swaps ggerganov/llama.cpp#3228
• llama : add support for batched inference ggerganov/llama.cpp#2813

[yourbuddyconner]

Also, looks like easy integration here is potentially contingent on this issue: ggerganov/llama.cpp#3478

[akx]

See #771 – it's one of the pinned issues in this repo.

[yugaljain1999]

@abetlen How to run batch inference of T5 model with llama-cpp-python?
Can you pls share correct code snippet to do that?

As per docs, if i run following code,

model= Llama(
      model_path="../madlad400-3b-mt.gguf",
      n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)


prompt = ["<2en> Wie kannst du es wagen, so mit mir zu reden, Bruder?", "<2en> Wie kannst du es wagen, so mit mir zu reden, Bruder?"]
prompt_tokenized = model.tokenize(prompt[0].encode())
print("Prompt tokenized", prompt_tokenized)


output_tokens = model.generate(
      prompt_tokenized, # encoded and tokenized prompt
      ) # Generate a completion, can also call create_completion

print("Output tokens", output_tokens)


for token in output_tokens:                     # -> Here I get error while iterating generator object.
    detokenized_tokens = model.detokenize([token])

This error -
GGML_ASSERT(n_outputs_enc > 0 && "call llama_encode() first") failed

[SylvainVerdy]

Did you find any solutions? @yugaljain1999

[yugaljain1999]

@SylvainVerdy Not yet, so I started using ctranslate2 for batch inference.