Post your hardware specs here if you got it to work. 🛠 #79
Comments
elephantpanda
changed the title
|
7B takes about 14gb of Vram to inference, and the 65B needs a cluster with a total of just shy of 250gb Vram The 7b model also takes about 14gb of system ram, and that seems to exceed the capacity of free colab, if anyone requires that. |
|
7B model passed under the following environment: With some modification: model_args: ModelArgs = ModelArgs(max_seq_len=1024, max_batch_size=1, **params) #
model = Transformer(model_args).cuda().half() # some people say it doesn't help
prompts = ["What is the most famous equation from this theory?"]
|
|
@Urammar could you also post how much Vram the other 2 Models need? I feel like this could help a lot of people to know what their machine can actually support. I only have a single A100 40GB and can therfore only run the 7B parameters model atm... 😅 |
|
Not sure if this will be helpful, but I made a spreadsheet to calculate the memory requirements for each model size, following the FAQ and Paper. You can make a copy to adjust the batch size and sequence length Will update as necessary |
|
How much VRAM does the 7B model need for finetuning? Are the released weights 32-bits? |
|
I just made enough code changes to run the 7B model on the CPU. That involved
I also set Steady state memory usage is <14GB (but it did use something like 30 while loading the model). It took 7.75 seconds to load the model (some memory swapping occurred during this so it may not be representative), 183 seconds to generate the first token, and 23 seconds to generate each token thereafter. It's only using a single CPU core for some reason (that I haven't tracked down yet). Hardware: Ryzen 5800x, 32 GB ram |
Can I ask you the biggest favor and provide your example.py file? :) |
This is probably what you want (the changes aren't just in example.py): https://github.com/gmorenz/llama/tree/cpu |
|
Gotcha. So all we would run is
|
Is more like it... also remove the extra prompts in the hardcoded prompts array. Also reduce max_gen_len if you want it to take less than 1.6 hours (but I just let that part run). |
|
I was able to run 7B on two 1080 Ti (only inference). Next, I'll try 13B and 33B. It still needs refining but it works! I forked LLaMA here: https://github.com/modular-ml/wrapyfi-examples_llama and have a readme with the instructions on how to do it: LLaMA with WrapyfiWrapyfi enables distributing LLaMA (inference only) on multiple GPUs/machines, each with less than 16GB VRAM currently distributes on two cards only using ZeroMQ. Will support flexible distribution soon! This approach has only been tested on 7B model for now, using Ubuntu 20.04 with two 1080 Tis. Testing 13B/30B models soon! How to?
|
|
With this code I'm able to run the 7B model on
All the code is doing is storing the weights on the CPU and moving them to the GPU just before they're used (and then back. Ideally we'd just copy them to the GPU and then never move them back... but I think that will take a more extensive change to the code). |
|
My results are in just to prove it works with only 12GB system ram! #105 Model 7B Took about a minute to load the model, it was maxing out the RAM and chomping on the page file. 😉 Loaded model in 116.71 seconds. Changes I made to example.py |
Hardware:
Llama 13B on a single RTX 3090In case you haven't seen it: I managed to get Llama 13B to run with it on a single RTX 3090 with Linux! Make sure not to install bitsandbytes from pip, install it from github! With 32GB RAM and 32GB swap, quantizing took 1 minute and loading took 133 seconds. Peak GPU usage was 17269MiB. Kudos @tloen! 🎉 Llama 7BSoftware:
What i had to do to get it (7B) to work on Windows:
Loading the model takes 5.1 seconds. On Ubuntu Linux 22.04.2 i was able to run the example with torchrun without any changes. Loading the model from an NTFS partition is a bit slower at 6.7 seconds and memory usage was 22916MiB / 24576MiB. nvidia drivers 530.30.02, CUDA 12.1. |
|
I have a version working with a batch size of 16 on a 2080 (8GB) using the 7B model
And from that I get around half an hour for 16 outputs of 512 length. It seemed like the average was 3 seconds per forward pass at 16 batch size. The most random output for me so far has been a bunch of floor related negative tweets, which came from the tweet sentiment analysis prompt |
|
@venuatu - check out my code for how I avoided doing a I'm also curious why you're doing |
|
Yep, that's a much better way to do it. It's now running in half the time (ty @gmorenz ) the |
|
I found some fixes for the very slow load times and its now down to 2.5 seconds (with a hot file cache) from my previous 83 seconds
|
|
Apple Silicon M1, CPU mode |
|
Specs: Ryzen 5600x, 16 gigs of ram, RTX 3060 12gb With @venuatu 's fork and the 7B model im getting: 46.7 seconds to load. |
|
My Specs: Using 7B, model loading time 5,61 sec used @gmorenz's fork which enable my tiny GPU to run it :) and changed from nccl to gloo
|
|
I got finally the 65GB model running on a server of the genesiscloud with 8 RTX 3090 cards with 24GB memory each. Takes almost 3 minutes to load. Inference is quicker than I can read. So far I am not impressed. I believe GPT-3 (text-davinci-002) is better. But I have to do more tests with different temperatures etc. Here is the result of one experiment: Why General Artifical Intelligence will overtake the world soon. An Essay by Llama. |
|
Hello guys, |
|
I have the 65B (120GB) model working at 60 seconds/token on: SSD read speed is (of course) the bottleneck - I'm just loading every layer from disk before using it and freeing all the memory (RAM and VRAM) afterwards. Will clean up the code and push it tomorrow. Goes without saying that at 60 seconds/token the utility of this is... questionable. |
I tried 7B with the CPU version on a M2 Max with 64GB ram, it's slow as heck but it works! Load time around 84secs and takes about 4mins to generate a response with Input:
Output:
Edit: on a 2nd try, the model load time is reduced to 34secs, not sure what changed, but keep in mind I'm running this in a Docker container (using |
|
Anyone have info regarding use with AMD GPUs? The 7b LLaMa model loads and accepts up to 2048 context tokens on my RX 6800xt 16gb I keep seeing people talking about VRAM requirements when running in 8 bit mode and no one's talking about normal 16 bit mode lol |
|
Got 7B loaded on 2x 8GB 3060's, using Kobold United, the dev branch, getting about 3 tokens/second. terbo: what is life? |
Does CUDA work on AMD? Someone tried to made a DirectML port: #117 Which should work on AMD (for Windows) but it hasn't been tested so it might need some fixing. |
|
I'm looking for the best laptop for the job. So what do you think about 64gb ram and 16gb vram ? |
Lol, laptop will just thermal throttle after 2min |
What are the "ideal" specs to run 65B on a PC?Is it possible to build a box with Llama 65B running in a process, that can still perform well as your daily driver? |
|
Okay, what about minimum requirements? What kind of model can run on old servers and how much RAM needed for just only run LLAMA2? |
|
Trained with SFTTrainer and QLora on Google Colab:
BitAndBytes (double quantize), Mixed Precision training (fp16="02") and gradient+batch sizes of 2 or lower helped out with memory constrains. If you don't have your own hardware, use Google Colab. This is a good starter: https://colab.research.google.com/drive/12dVqXZMIVxGI0uutU6HG9RWbWPXL3vts |
|
Do you have a tutorial/ video... ? |
|
I have an M1 MacBook Pro 16" with 16GB ram. it runs both the 7B and 13B models. Loads with no delay and I usually get instant response from both though additional info can take around 5 seconds to appear. They appear to have a wild imagination when it comes to accuracy so take most answers with pinch of salt. Sometimes after asking an initial question It often goes off and starts asking its own questions and then answers itself. I suppose it is presuming these are standard follow up questions that most people will ask. The 13B model can become unstable after some use. usually get a load of repeating text then it locks up. |
|
Llama2 7B-Chat on RTX 2070S with bitsandbytes FP4, Ryzen 5 3600, 32GB RAM Completely loaded on VRAM ~6300MB, took ~12 seconds to process ~2200 tokens & generate a summary(~30 tokens/sec). Llama.cpp for llama2-7b-chat (q4) on M1 Pro works with ~2GB RAM, 17tok/sec Also ran the same on A10(24GB VRAM)/LambdaLabs VM with similar results |
|
Llama2 7B-Chat official sample (with exactly the same launching arguments in README) GPU: 4060Ti 16GB. Consumes more than 14GB torchrun --nproc_per_node 1 example_chat_completion.py --ckpt_dir llama-2-7b-chat/ --tokenizer_path tokenizer.model --max_seq_len 512 --max_batch_size 4 |
|
Task: Fine tune Llama2 7B and 13B on a task specific function using my own
data
GPU: 3090 24GB
RAM : 256 GB
CPU: 3970X
I have two GPUs but I only wanted to use one so I ran the following in my
terminal so the script could only see the first GPU in my system
export CUDA_VISIBLE_DEVICES=0
I trained with LORA rank of 32, batch size 1, context length of 4096.
After training for 2000 steps I saw a noticeable improvement on the task I
was training for, loss went from ~1.8/1.4 for 7B/13B base models to 0.41 /
0.33 after 5000steps and I still have room to go (0.5 through an epoch).
The task I'm training on is the recognition and description of malicious
decompiled code (malware).
…On Fri, Jul 28, 2023 at 10:16 AM K.G. Wang ***@***.***> wrote:
Llama2 7B-Chat official sample (with exactly the same launching arguments
in README)
GPU: 4060Ti 16GB. Consumes more than 14GB
RAM: 16GB. The memory usage is about 2GB after the model is loaded
torchrun --nproc_per_node 1 example_chat_completion.py --ckpt_dir llama-2-7b-chat/ --tokenizer_path tokenizer.model --max_seq_len 512 --max_batch_size 4
—
Reply to this email directly, view it on GitHub
<#79 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABNELIQ5PBPFP575SMHELT3XSPCVPANCNFSM6AAAAAAVODAZWY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Llama 7B and 13B both GGML quantized. Hardware:
Running in local (no huggingface, etc) with LlamaCpp |
250GB for a 65b model seems a bit too much, I think most of the examples out there for 65b model usually about 140GB is needed. Any insight on the reason for the difference? Is it fine tuning memory usage? |
Are you loading it in full-precision, i.e., |
Wait, I thought Llama was trained in 16 bits to begin with. |
That is true, but you will still have to specify the dtype when loading the model otherwise it will default to |
Ah I see what you meant. Thanks for clarification. I was using hugging face version with their transformers package, so I guess that was the reason I didn’t see such a big memory usage. But seems a waste of memory to cast 16 bit model to 32 bit? Is there any reason you kept the PyTorch default precision? |
I can't comment on design decisions made by Huggingface but I stick to specifying the dtype regardless of the model I load. |
Apologies. Apparently I mistook you as the original comment author. Yes, I’d agree specifying dtype to match the native precision of the model would be a good idea. |
|
Thanks to all for the data. In English (MT, not edited) ChatGPT Humanity developed and progressed for thousands of years. It invented the wheel, learned to work with metals, invented writing, then printing, the mechanical calculator, the telegraph, the telephone and the radio, television, and electronic computers. People wanted to preserve their knowledge for future generations, they wanted to communicate over long distances, to have "eyes" everywhere in the world. All of this became a reality thanks to human genius and the desire of Homo sapiens to control information. In his pursuit, he created the universal machine for executing algorithms, for processing information - the electronic computer. The idea for it occurred to John Atanasoff in the late thirties, and it became a reality in 1945 in the USA under the name "ENIAC." Like most major discoveries, the computer was initially used only for military purposes, but later it found its rightful place in almost all spheres of our life. Computing machines developed rapidly - from electronic tubes through transistors and to the highest technology - integrated circuits, which allowed electronic engineers to create modern microprocessors with millions of transistors that fit in the palm of a hand, capable of performing billions of calculations per second. The impact of the global computer network, the Internet, on people today is palpable. The network will continue to expand, but I believe that this is not the culmination of the rapid development of electronics, computing technology, and software (the latter actually determines the computer's "behavior," i.e., it is its "soul," and electronics is part of its material form). Another discovery, still unrealized, will crown this progress - artificial intelligence. Some people believe that computers are "soulless machines" doomed to remain only assistants to human intelligence and creativity, but not to become creators themselves and possess reason. Machines are created by humans, I believe that he is capable of creating a system, an algorithm that imitates the work of his brain, and in this way, the computer can become a thinking machine. In my opinion, the path of humanity leads there - towards the creation of an electronic analog of the highest creation of nature, what is currently unique to humans - intelligence. The creation of artificial intelligence will change the world. In my opinion, artificial intelligence is the next step in the evolution of matter - computers have several advantages over the "transitional" human, the most essential of which is that they are practically immortal - they withstand all kinds of radiation, do not feel pain, require very little energy, which they can easily obtain from the Sun, and they can reproduce themselves by producing factories, and so on. My opinion is that the world is heading precisely there, towards the creation of the thinking machine - the machine of God.
|
|
CodeLLAMA-13B Running on RTX3090. |
|
@hz-nm, how fast is it in terms of tokens per second? |
Now I'm stuck with 5 questions... what else are you running on that machine? I feel like there needs to be some kind of helper app in there. Maybe I need a better understanding of the hardware needs... can I run it on CPU alone if I have enough CPU memory? Can I run these larger models on a regular PC? Can I get a few P40s or K40s and offload certain tasks to this? I'm really curious about the hardware needs for running these models... |
I didn't put a measure for that unfortunately but it is quite fast. Almost as good as ChatGPT if it was streaming. |
|
Hey guys, I want to deploy code Llama on a Ubuntu server specifically on the cloud, what specs should I use like the vCPU and memory? Please suggest or guide for the same. Thanks in advance. |
|
Am new to this and in testing phase to see what works. |
and others
It might be useful if you get the model to work to write down the model (e.g. 7B) and the hardware you got it to run on. Then people can get an idea of what will be the minimum specs. I'd also be interested to know. 😀
The text was updated successfully, but these errors were encountered: