[Q] Memory Requirements for Different Model Sizes

[NightMachinery]

No description provided.

[satyajitghana]

7B (4-bit): 4.14 GB MEM
65B (4-bit): 38 GB MEM

[prusnak]

Since the original models are using FP16 and llama.cpp quantizes to 4-bit, the memory requirements are around 4 times smaller than the original:

• 7B => ~4 GB
• 13B => ~8 GB
• 30B => ~16 GB
• 65B => ~32 GB

[cannin]

With an M1 Max 64GB with 4-bit

65B: 38.5GB, 850 ms per token
30B: 19.5GB, 450 ms per token
13B: 7.8GB, 150 ms per token
7B: 4.0GB, 75 ms per token

[dbddv01]

For the record, Intel® Core™ i5-7600K CPU @ 3.80GHz × 4, 16Gb ram, under Ubuntu, model 13B runs with acceptable response time. Note that as mentioned by previous comments, -t 4 parameter gives the best results.
main: mem per token = 22357508 bytes
main: load time = 83076.67 ms
main: sample time = 267.12 ms
main: predict time = 193441.61 ms / 367.76 ms per token
main: total time = 277980.41 ms

Great work !

[ggerganov]

Should add these to readme

[sinanisler]

@prusnak is that pc ram or gpu vram ?

[prusnak]

@prusnak is that pc ram or gpu vram ?

llama.cpp runs on cpu not gpu, so it's the pc ram

[whitepapercg]

@prusnak is that pc ram or gpu vram ?

llama.cpp runs on cpu not gpu, so it's the pc ram

Is it possible that at some point we will get a video card version?

[prusnak]

Is it possible that at some point we will get a video card version?

I don' think so. You can use run the original Whisper model on a GPU: https://github.com/openai/whisper

[mrpher]

Fwiw, running on my M2 Macbook Air w 8GB of ram comes to a grinding halt. At first run about 2-3 minutes of completely unresponsive machine (mouse and keyboard locked), then about 10-20 seconds per response word. Didn't expect great response times, but thats a bit slower than anticipated.

• Edit: using 7B model

[prusnak]

M2 Macbook Air w 8GB

Close every other app, ideally reboot to clean state. This should help. If you see unresponsive machine, then it is swapping memory to disk. 8GB is not that much, especially if you have Browsers, Slack, etc. running.

[j-f1]

Also make sure you’re using 4 threads instead of 8 — you don’t want to be using any of the 4 efficiency cores.

[mrpher]

Working well now, good recommendations @prusnak @j-f1 thank you

[prusnak]

Requirements added in #269

[SpeedyCraftah]

Since the original models are using FP16 and llama.cpp quantizes to 4-bit, the memory requirements are around 4 times smaller than the original:

• 7B => ~4 GB
• 13B => ~8 GB
• 30B => ~16 GB
• 64 => ~32 GB

32gb is probably a little too optimistic, I have DDR4 32gb clocked at 3600mhz and it generates each token every 2 minutes.

[prusnak]

32gb is probably a little too optimistic

Yeah, 38.5 GB is more realistic.

See https://github.com/ggerganov/llama.cpp#memorydisk-requirements for current values

[SpeedyCraftah]

32gb is probably a little too optimistic

Yeah, 38.5 GB is more realistic.

See https://github.com/ggerganov/llama.cpp#memorydisk-requirements for current values

I see. That makes more sense since you mention the whole model is loaded into memory as of now. Linux would probably run better in this case from the better swap handling and lower memory usage.

Thanks!

[Yitzhokchaim]

What languages ​​does it work with? Does it work in the same output and input languages ​​GPT?