Implementations for Q4_0_8_8 quantization based functions in AVX2 SIMD architecture #8713
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ggml
Srihari-mcw
commented
Jul 26, 2024
ggml/src/ggml-aarch64.c
Outdated
| __m256i requiredOrder = _mm256_set_epi32(3 ,2 ,1 ,0, 7 ,6, 5, 4); | ||
|
|
||
| // Take group of four block_q8_0x4 structures at each pass of the loop and perform dot product operation | ||
| for (; y < nr / 4; y += 4) { |
There was a problem hiding this comment.
ne11 is processed in batches of 16 in GEMM function. Leftover ne11 is processed in batches of four. Saw higher boostup in performance while processing ne11 in batches of 16 and leftover in batches of 4 versus just processing ne11 in batches of four
|
this isn't a new conversion type, right? it's just a new way of calculating Q4_0? |
|
Hi @bartowski1182 , the Q4_0_8_8 is a format of quantization where the values are stored in the same 4 bit quantized fomat, along with the same delta values as Q4_0. The 4 bit quantized quants values across eight different blocks are interleaved with each other. This was introduced in PR #5780 . Models that needs to use this particular code path, needs to be quantized in this particular format of Q4_0_8_8. Thanks |
|
I just tested this. It works. Albeit nowhere as drastically as it helps on ARM cpus but it helps , tps on intel cpu inference of 4bit llama 405B went from 0.78 (meta-llama-405b-Q_4_NL) to 0.89 meta-llama-405b-Q_4_0_8_8 . the q4nl and q4088 shape filesize are identital. prompt: Again both x86 and ARM cpus max out batch-size-1 inference at 64 cores or threads, any more and it slows down a bit or stays stagnant. |
mofosyne
added
the
Review Complexity : Medium
The main benefit from these changes should be in the prompt processing speed, not the text generation. Better to use |
ggerganov
approved these changes
Sep 3, 2024
There was a problem hiding this comment.
@Srihari-mcw Could you make a perplexity comparison before merging? For example against a Q4_0 model at PPL 32 chunks
…8x8_q8_0 and ggml_gemm_q4_0_8x8_q8_0 functions
…o be processed as multiple of 16 in MSVC
Srihari-mcw
force-pushed
the
block_interleaving_q4_0_8_8_avx2_implementation
branch
from
81d9078
to
c950fc3
Compare
|
Hi @ggerganov, The perplexity was measured for models quantized from meta llama2 7B model with the following command : It calculated perplexity over 655 chunks : The perplexity results are tabulated as follows :
The perplexity readings were found to be almost the same post the tests Further, post the latest changes in master branch and in the PR, the performance readings are as follows GCC Linux : Q4_0 Model :
GCC Version = 12.3 The PR was tested in AMD Raphael 7600X which supports the following flags by default : AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1| Thanks |
|
It looks like the rocm compiler is crashing when compiling this code, which is breaking the generation of docker images. |
dsx1986
pushed a commit
to dsx1986/llama.cpp
that referenced
this pull request
Oct 29, 2024
* Add AVX2 based implementations for quantize_q8_0_4x8, ggml_gemv_q4_0_8x8_q8_0 and ggml_gemm_q4_0_8x8_q8_0 functions * Update code to fix issues occuring due to non alignment of elements to be processed as multiple of 16 in MSVC * Update comments and indentation * Make updates to reduce number of load instructions
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Nov 15, 2024
* Add AVX2 based implementations for quantize_q8_0_4x8, ggml_gemv_q4_0_8x8_q8_0 and ggml_gemm_q4_0_8x8_q8_0 functions * Update code to fix issues occuring due to non alignment of elements to be processed as multiple of 16 in MSVC * Update comments and indentation * Make updates to reduce number of load instructions
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Nov 18, 2024
* Add AVX2 based implementations for quantize_q8_0_4x8, ggml_gemv_q4_0_8x8_q8_0 and ggml_gemm_q4_0_8x8_q8_0 functions * Update code to fix issues occuring due to non alignment of elements to be processed as multiple of 16 in MSVC * Update comments and indentation * Make updates to reduce number of load instructions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GCC Linux :
Q4_0 Model :
The models were quantized and tested from meta-llama2 7B model - https://huggingface.co/meta-llama/Llama-2-7b
The PR was tested in AMD Raphael 7600X which supports the following flags by default :
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1|