Code

llama : ggml-backend integration#4766

Code

Merged

slaren merged 39 commits intomasterggerganov/llama.cpp:masterfrom sl/backend-schedggerganov/llama.cpp:sl/backend-schedCopy head branch name to clipboardJan 12, 2024

+2,523-2,285

Commits

Commits on Jan 6, 2024

• llama : ggml-backend integration

  slaren

  committed
  33f0761

• ggml-backend : add names to buffers

  slaren

  committed
  6483328

• fix unmap after loading

  slaren

  committed
  a1ab35c

• batched-bench : add tensor_split param

  

  ggerganov

  authored and

  slaren

  committed
  1fa7ee2

• llama : check for null tensor_split

  slaren

  committed
  863ef45

• ggml-backend : increase GGML_MAX_BACKENDS

  slaren

  committed
  d107459

• improve graph splitting, partial fix for --no-kv-offload

  slaren

  committed
  ece0b0d

• cuda : add ggml-backend split buffer support

  slaren

  committed
  2f2c367

• cuda : do not create buffer types for devices that don't exist (fixes usage without CUDA devices available)

  slaren

  committed
  72b74f3

Commits on Jan 7, 2024

• ggml : fix null backend dereference (#4807)

  Show description for f77c72f
  ggerganov

  authored
  f77c72f

• test-backend-ops : check buffer allocation failures

  slaren

  committed
  7c16cf1

• Merge remote-tracking branch 'origin/master' into sl/backend-sched

  slaren

  committed
  87c8207

• llama : add cparam (split_mode) and command line argument (--split-mode, -sm) to configure the split mode (none, layer or row)

  slaren

  committed
  5e879c9

Commits on Jan 8, 2024

• ggml : fix mul_mat_id work size

  slaren

  committed
  ac145fd

• llama : rewrite session kv load/set without graphs

  slaren

  committed
  444b975

• minor

  slaren

  committed
  d41cef9

• llama : only initialize used backends, free backends on context free

  slaren

  committed
  5a62db3

• llama : abort ctx if cuda backend init fails

  slaren

  committed
  4813e17

• llama : rewrite lora with ggml-backend and compute on CPU

  Show description for 11583c1
  slaren

  committed
  11583c1

• llama : only map to a backend buffer the region of the file mapping containing the tensors used in the buffer

  slaren

  committed
  4ed5f62

Commits on Jan 9, 2024

• opencl : add ggml-backend buffer type

  slaren

  committed
  fa76201

• Merge remote-tracking branch 'origin/master' into sl/backend-sched

  Show description for 2e7814a
  slaren

  committed
  2e7814a

Commits on Jan 10, 2024

• cuda : only use batched_cublas with batched mat muls (fixes fp16 tg perf)

  slaren

  committed
  5d2dffc

• Merge remote-tracking branch 'origin/master' into sl/backend-sched

  slaren

  committed
  3cb1c1f

• llama : on Metal, by default offload the full model

  Show description for 07a1b05
  ggerganov

  committed
  07a1b05

• metal : page align the data ptr (#4854)

  ggerganov

  authored
  3cd0cbb

• Apply suggestions from code review

  Show description for 74066f8
  

  slaren

  and

  JohannesGaessler

  authored
  74066f8

• cuda : fix split buffer free

  slaren

  committed
  c522c11

Commits on Jan 11, 2024

• address review comments

  slaren

  committed
  9d4ba6e

• llama-bench : add split-mode parameter

  slaren

  committed
  d83c084

• fix whitespace

  slaren

  committed
  6dcc42b

• opencl : fix double initialization

  slaren

  committed
  42aa835

• Merge remote-tracking branch 'origin/master' into sl/backend-sched

  slaren

  committed
  c3681af

• server : add --split-mode parameter

  slaren

  committed
  c486719

Commits on Jan 12, 2024

• use async copy and compute to improve multi-gpu performance

  Show description for 23c14ef
  slaren

  committed
  23c14ef

• use async memcpys to copy the graph outputs to the CPU

  slaren

  committed
  e73009e

• fix opencl

  slaren

  committed
  1e7694e

• Merge remote-tracking branch 'origin/master' into sl/backend-sched

  slaren

  committed
  458674c

• use a host buffer for the cpu compute buffer for faster copies to the gpu

  slaren

  committed
  53ae0dd