CUDA : faster k-quant dot kernels

[ikawrakow]

After the CUDA inference speed up in PR #1827 I got motivated to improve the performance of the k-quants, and this PR is the result.

The following table compares token prediction times on current master (i.e., after merging #1827) and this PR for all k-quantization levels. Shown is the average of 3 runs in milliseconds predicting 128 tokens on an RTX-4080 GPU using the following command:

./bin/main -m ../models/7B/qXk.bin -p "I believe the meaning of life is" -c 2048 -n 512 --ignore-eos -n 128 -s 1234 -t 1 -ngl 100

Quantization	Master	This PR	Speedup
Q2_K	11.94	10.62	12.4%
Q3_K_S	15.35	11.68	31.4%
Q4_K_S	12.09	10.98	10.1%
Q5_K_S	13.53	12.03	12.5%
Q6_K	14.51	13.33	8.9%
Q4_0	12.40	-	-

I have added Q4_0, not touched by this PR, for comparison. It is now more than 10% slower than Q4_K, bit it should be faster as it needs to do less work, so there is a potential for improvement there (on Metal, Q4_0 is ~10% faster than Q4_K).

I have moved away from using a templated implementation. There are subtle differences between the different k-quants in what is the best way to spread the threads in a thread group within a quantization block, and these variations are difficult to capture in a template. The result is ~100 extra lines of code compared to the templated implementation, which I think is worth it, given the increase in performance.

[JohannesGaessler]

I didn't look at the k-quant code until now because I was preoccupied with other things. The code added in this PR looks good to me assuming it works as it should. I will advise you to be cautious with GPU performance optimizations though. Optimal values for e.g. the block size are not the same across different cards; it's possible that this PR causes a performance regression for some GPUs.

[JohannesGaessler]

You could just pass 32 directly instead of block_dims which I think would be a little cleaner since it avoids mixing scalars with three-dimensional vectors.

[JohannesGaessler]

I did some performance testing:

GPU	Model	ms/t master	ms/t PR
RTX 3090	7b q2_K	17.24	16.49
RTX 3090	7b q3_K_S	21.31	18.42
RTX 3090	7b q4_K_S	17.74	17.11
RTX 3090	7b q5_K_S	18.84	17.98
RTX 3090	7b q6_K	18.76	17.62
RTX 3090	7b q4_0	15.76	-
GTX 1070	7b q2_K	91.87	127.91
GTX 1070	7b q3_K_S	91.59	81.96
GTX 1070	7b q4_K_S	129.12	134.02
GTX 1070	7b q5_K_S	144.76	136.66
GTX 1070	7b q6_K	81.73	143.29
GTX 1070	7b q4_0	62.61	-

On my RTX 3090 the new kernels are consistently faster. q4_0 is still faster though (maybe you didn't compile with LLAMA_CUDA_DMMV_X=64 LLAMA_CUDA_DMMV_Y=2?). On my GTX 1070 the performance is only better for two out of the five quantization formats. For q2_K and q6_K the performance is significantly worse.

[slaren]

It would be great if the performance of q2_K and q6_K on older GPUs can be fixed, but if you don't have a Pascal GPU for testing, I think it is ok prioritizing the performance on newer architectures.

[ikawrakow]

@JohannesGaessler Thank you for taking the time to look into this PR and posting results for different cards. Do you have a hypothesis why the new version tuns slower on the 1070?

You are right, i did not use LLAMA_CUDA_DMMV_X=64 LLAMA_CUDA_DMMV_Y= 2 for the Q4_0 result. Using this, Q4_0 token prediction time becomes 11.5 ms on my 4080 card, so still slower than Q4_K_S.

[JohannesGaessler]

It would be great if the performance of q2_K and q6_K on older GPUs can be fixed, but if you don't have a Pascal GPU for testing, I think it is ok prioritizing the performance on newer architectures.

I personally would not merge an optimization that causes a small performance uplift for some users but a large performance malus for other users but I am neither the project lead nor the developer of k-quants so I'll leave the decision up to @ikawrakow

Do you have a hypothesis why the new version tuns slower on the 1070?

I don't. GPU performance optimizations depend very strongly on the hardware details but the difference between an RTX 3090 and a GTX 1070 is quite large. It oftentimes comes down to something like the number of warps that fit into the available registers on a streaming multiprocessor and it's much easier to just tweak parameters and test performance than to try to understand what's happening on the hardware side. It's also possible that you are now using some feature that just has bad performance on older cards.

[ggerganov]

I'm also leaning towards improving the performance on newer devices and optionally having compile flags / macros for better performance on older hardware given that this does not complicate or duplicate the code too much (i.e. similar to the GGML_CUDA_DMMV_X, GGML_CUDA_DMMV_Y defines).

But before making any decision, let's wait for one or two more reports that confirm the degraded performance on older cards, just in case

[ikawrakow]

I did get my hands on a box with a GTX-1660 (released Oct 2019). Q3_K, Q4_K and Q5_K are faster with this PR, Q2_K and Q6_K are slower. So, I guess, I'll do something with ifdefs for Q2_K and Q6_K.

If I remember correctly, back then the 1660 was basically the cheapest option if one wanted to have a discrete GPU but didn't want to pay the premium Nvidia added with the introduction of the 2000 series. The card has 6 GB of VRAM and the 4-bit models fail to run on it with -ngl 100 on current master. They run with -ngl 32, but performance is lower compared to what we had before #1827 was merged, so it seems we merged a PR without being too much worried about the experience of users on older hardware.

I also got my hands on an even older PC gathering dust in the corner. It has a 1060 card in it, so I wanted to experiment with that. But after fighting with Nvidia drivers and CUDA toolkits for a while, I eventually gave up. Are we aware that none of the CUDA stuff will work with the default packages on Ubuntu 20.04 LTS? I any case, while at it, I ran Q4_K_S on the CPU (i9-9960X, released 2018) and it was faster than the numbers @JohannesGaessler posted for the 1070 card.

[johnson442]

Some more data from the type of hardware in question - 1070ti and i5-8600K:

	Master	PR	CPU (-t 5)
7B
Q2_K	84.92	116.34	83.68
Q3_K_S	83.07	75.21	86.10
Q4_K_S	120.97	122.55	102.77
Q5_K_S	136.51	127.96	122.62
Q6_K	74.18	131.62	143.83
Q4_0	57.50	-	103.09
13B (-ngl 24 -t 5)
Q6_K	196.96	255.84	273.61

I've included CPU only (with 5 threads on 6 cores) to compare, as well as 24 layers of a 13B model on the GPU (max in 8GB of VRAM).

Forgive me if its explained elsewhere, but is there a reason Q6_K is faster than the other new quants on this GPU?

[maddes8cht]

With my rtx 3060 using the #1827 build performance got actually worse when using the "extra layers" with 13b models.
Generation times got about 40% slower, and prompt eval time raised about 25 times up ( from something like 80 ms/token to 2000 ms/token)
Going back to -ngl 40 with 13b models braught me back a lot of performance.

How about making block sizes configurable at run time?
And maybe have a small helper app that tests out optimal parameters?

[ikawrakow]

Forgive me if its explained elsewhere, but is there a reason Q6_K is faster than the other new quants on this GPU?

The Q6_K quantization is simpler. The weight w is obtained from the quant q as w = d * s * q, where d is the scale of the super-block and s is the scale of the block inside the super-block in which q is located. In Q6_K the scale is 8 bit, so it can be taken and used directly. Q3_K is similar, but there the scale is 6 bit, so one needs more compute (bit manipulations) to recover it from the densely stored data. For the other k-quants we have w = d * s_q * q - m * s_m, where now m is another scale for the super-block and s_q, s_m are per-block 6-bit parameters (4-bit in the case of Q2_K). As a result one needs way more bit fiddling to recover the scales and the quants.

When the computation becomes very fast as it is the case on the latest generation Nvidia cards, memory bandwidth becomes important, and this results in Q6_K being slower than the others on my RTX-4080. To give more specific numbers, we have Q6_K taking ~13.5 ms. If I put return as the very first statement in the dot product kernel, the computation completes in ~3.5 ms/token. I.e., the dot product takes ~10 ms. The Q6_K 7B model is 5.5 GB, so this means ~550 GB/s are being read from memory while the dot products are being computed. The RTX-4080 spec claims 768 GB/s memory bandwidth, but that would be under ideal conditions with all threads/warps operating in sync and fetching data from memory in the optimum way. We don't have that here as we are reading single bytes and jumping around within the data of a super-block to fetch scales/quants, so my guess is that 550 GB/s is as good as it gets (or at least very close to). In comparison, Q4_K_S runs in ~11 ms, so ~7.5 ms for dot products. The model is 3.8 GB, so ~500 GB/s are being read from VRAM while running inference with a Q4_K model.

[JohannesGaessler]

@maddes8cht I saw some people complain about how with the newest NVIDIA drivers the program no longer crashes when you run out of VRAM. Instead data from VRAM gets moved to RAM and you get terrible performance. And it's a "feature" that cannot be disabled.

[ikawrakow]

I put some effort into improving the performance of Q2_K and Q6_K on older GPUs, while preserving performance on newer cards. The oldest GPU I have available is a GTX-1660 released in 2019, so I would appreciate if someone tested on the the GTX-1000 series. To deal with older cards, I have added LLAMA_CUDA_KQUANTS_ITER to CMakeLists.txt and Makefile. It is set by default to 2 (best for RTX-4080). On older cards performance is better with LLAMA_CUDA_KQUANTS_ITER=1.

Here are the per token prediction times for 128 tokens using the 7B model after the last commits on a GTX-1660 using LLAMA_CUDA_KQUANTS_ITER=1 (performance on the RTX-4080 is the same as given in the initial description of the PR):

Quantization	Master	This PR	Speedup
Q2_K	52.2	47.4	10.1%
Q3_K_S	65.6	60.3	8.8%
Q4_K_S	68.0	65.5	3.8%
Q5_K_S	74.4	68.5	8.6%
Q6_K	63.0	62.5	0.9%
Q4_0	50.6	-	-
Q5_1	87.6	-	-

The GTX-1660 has 6 GB of VRAM, so only Q2_K and Q3_K fit completely on the GPU (including KV cache) and are run with -ngl 100 -t 1. Q4_K and Q5_K were run with -ngl 32 -t 4, Q6_K with -ngl 29 -t 4, and Q5_1 with -ngl 30 -t 4.

[johnson442]

On a 1070ti:

	Master (ms/t)	New PR	New PR ITER=1
7B
Q2_K	84.92	91.13	76.54
Q3_K_S	83.07	74.76	73.63
Q4_K_S	120.97	122.39	121.69
Q5_K_S	136.51	127.34	126.65
Q6_K	74.18	133.55	73.28
13B (-ngl 24 -t 5)
Q6_K	196.96	260.25	198.73

Faster than master in most cases, amazing stuff!

And thanks as well for the detailed explanation of Q6_K.

[johnson442]

Should something be added to the README maybe, either a small CUDA performance section or just added to the CUBLAS instructions - mentioning LLAMA_CUDA_DMMV_X/Y and LLAMA_CUDA_KQUANTS_ITER?

I guess there is no K-quant stuff yet either, LLAMA_CUDA_KQUANTS_ITER might be best put in a future section describing them.

[mirek190]

I tested with RTX 3090 and 33B q4k_m molel with 63 layers on GPU.

Before your patch I had 89 ms/t and now 82ms/t

Great work!!

[dragonfyre13]

Dropping a comment here because I seem to have independently come across very similar results downstream in koboldcpp, was actually originally coming to submit a PR to llama.cpp to have a compile time check if compute capability <7, if so set K_QUANTS_PER_ITERATION to 1 vs 2. On my 3070 vs p40, modifying the compile flag LLAMA_CUDA_KQUANTS_ITER makes a drastic difference (2x) in ms/t for the p40. I'm headed over to open something in the kobold.cpp side, but figured it's worth dropping confirmation about the difference in this ticket as well.

Given it seems to make enough difference, does it make sense to have a check for pascal architecture somewhere? My CPP is a bit rusty at this stage, but I can cobble together a PR if it'd help. I'm not sure without digging, but seems like there's an opportunity for a runtime check even, if the system has both a compute capability <7 device and a >7 device (i.e. they have a 3070 and a p40)

[cmp-nct]

I've tested the update with Falcon 40 q6_k and it is a solid 6-7% boost in performance 92ms/tk went down to 86ms/tk for that large 38GB model. Awesome work!

[mirek190]

strange ... I tested latest build with rtx 3090 and 33B model q4k_m before i had around 82 ms/t and now I have 59 mn/s ....