ggml : Load data into int8x16x4_t using vld4q_s8 on arm64

[lindeer]

checkout latest 5c64a09@master, compilation report:

/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1262:43: error: invalid initializer
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1263:43: error: invalid initializer
             const int8x16x4_t q8bytes_2 = vld1q_s8_x4(q8); q8 += 64;
                                           ^~~~~~~~~~~
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c: In function ‘ggml_vec_dot_q5_K_q8_K’:
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1791:41: error: invalid initializer
             const int8x16x4_t q8bytes = vld1q_s8_x4(q8); q8 += 64;
                                         ^~~~~~~~~~~

but everything is OK on x86_64, maybe arm64 does not support this intrinsic?
Using vld4q_s8 instead of vld1q_u8_x4 seems working, both on x86_64 and arm64.
However, testing did not pass all due to issue #1736 .

$ make test
Running tests...
Test project /home/wesley/Work/projects/chatgpt/llama.cpp/build
    Start 1: test-quantize-fns
1/4 Test #1: test-quantize-fns ................   Passed    0.01 sec
    Start 2: test-quantize-perf
2/4 Test #2: test-quantize-perf ...............   Passed    0.01 sec
    Start 3: test-sampling
3/4 Test #3: test-sampling ....................Child aborted***Exception:   0.05 sec
    Start 4: test-tokenizer-0
4/4 Test #4: test-tokenizer-0 .................   Passed    0.02 sec

75% tests passed, 1 tests failed out of 4

Total Test time (real) =   0.09 sec

The following tests FAILED:
	  3 - test-sampling (Child aborted)
Errors while running CTest
make: *** [Makefile:84：test] 错误 8

[ggerganov]

@lindeer
This change breaks the inference on M1 Pro and M2 Max so I reverted it.
What hardware are you using?

[lindeer]

@ggerganov a debian linux with HUAWEI Kyrin 990
should I use macro? something like:

#if ARM
#define vld1q_s8_x4 vld4q_s8
#endif

[ikawrakow]

@lindeer No, vld4q_s8 is not the same as vld1q_s8_x4.

You need something like this:

#if DONT_HAVE_QUAD_LOAD
static inline int8x16x4 vld1q_s8_x4(const int8_t * address) {
    int8x16x4 result;
    result.val[0] = vld1q_s8(address);
    result.val[1] = vld1q_s8(address + 16);
    result.val[2] = vld1q_s8(address + 32);
    result.val[3] = vld1q_s8(address + 48);
    return result;
}
#endif

[lindeer]

thanks @ikawrakow , maybe there is something wrong with my usage. vld1q_s8_x4 is actually defined in my arm_neon.h(mine is __LITTLE_ENDIAN__):

#ifdef __LITTLE_ENDIAN__
#define vld1q_s8_x4(__p0) __extension__ ({ \
  int8x16x4_t __ret; \
  __builtin_neon_vld1q_x4_v(&__ret, __p0, 32); \
  __ret; \
})
#else
#define vld1q_s8_x4(__p0) __extension__ ({ \
  int8x16x4_t __ret; \
  __builtin_neon_vld1q_x4_v(&__ret, __p0, 32); \
 \
  __ret.val[0] = ...
  __ret; \
})
#endif

but in this context error is reported, I could not figure it out:

        for (int j = 0; j < QK_K/128; ++j) {

            const uint8x16x2_t q3bits = vld1q_u8_x2(q3); q3 += 32;
            const int8x16x4_t q8bytes_1 = vld1q_s8_x4(q8); q8 += 64;
            const int8x16x4_t q8bytes_2 = vld1q_s8_x4(q8); q8 += 64;




/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c: In function ‘ggml_vec_dot_q3_K_q8_K’:
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1262:43: warning: implicit declaration of function ‘vld1q_s8_x4’; did you mean ‘vld1q_s8_x2’? [-Wimplicit-function-declaration]
             const int8x16x4_t q8bytes_1 = vld1q_s8_x4(q8); q8 += 64;
                                           ^~~~~~~~~~~
                                           vld1q_s8_x2
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1262:43: error: invalid initializer
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1263:43: error: invalid initializer
             const int8x16x4_t q8bytes_2 = vld1q_s8_x4(q8); q8 += 64;
                                           ^~~~~~~~~~~
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c: In function ‘ggml_vec_dot_q5_K_q8_K’:
/home/wesley/Work/projects/chatgpt/llama.cpp/k_quants.c:1791:41: error: invalid initializer
             const int8x16x4_t q8bytes = vld1q_s8_x4(q8); q8 += 64;

while vld1q_u8_x2 and vld4q_s8 is OK:

#ifdef __LITTLE_ENDIAN__
#define vld1q_u8_x2(__p0) __extension__ ({ \
  uint8x16x2_t __ret; \
  __builtin_neon_vld1q_x2_v(&__ret, __p0, 48); \
  __ret; \
})


#ifdef __LITTLE_ENDIAN__
#define vld4q_s8(__p0) __extension__ ({ \
  int8x16x4_t __ret; \
  __builtin_neon_vld4q_v(&__ret, __p0, 32); \
  __ret; \
})

$ gcc --version
gcc (Uos 8.3.0.3-3+rebuild) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE