sycl : try to fix SYCL after IQ1_S changes #5995
Conversation
ggerganov
force-pushed
the
gg/try-fix-sycl-iq1_s
branch
from
8c9f7d9
to
77d586f
Compare
|
Could anyone with SYCL hardware run |
|
Thanks @ggerganov for adding the new changes , there are some changes needed on it. I will apply once i am near my laptop. |
ggml-sycl.cpp
Outdated
| const float delta = x[i].qh[ib] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA; | ||
| const float d = (float)x[i].d * (2*((x[i].qh[ib] >> 12) & 7) + 1); | ||
| uint32_t grid32[2]; const int8_t * q = (const int8_t *)grid32; | ||
| grid32[0] = iq1s_grid_gpu[x[i].qs[4*ib+il] | (((x[i].qh[ib] >> 3*il) & 7) << 8)]; |
There was a problem hiding this comment.
AFAICT, we can't use iq1s_grid_gpu constant table directly - that is why we pass the iq1s_grid pointer to this kernel.
Btw, there have been more changes to IQ1_S (#5999) since I opened this PR that should also be reflected. But I don't have means to do tests, so implementing this blindly is difficult
My suggestion is to push empty IQ1_S kernels on master so that the CI becomes green and your team can work on implementing the kernels and verifying that test-backend-ops runs correctly
There was a problem hiding this comment.
Agreed , let me try to build a solution. Thanks for the pointer. In the meantime, we can revert the initial solution using grid1 & 2
ggerganov
force-pushed
the
gg/try-fix-sycl-iq1_s
branch
from
af93a92
to
76be02a
Compare
* sycl : try to fix after IQ1_S changes * sycl : iq1s_grid -> iq1s_grid_gpu * sycl : fix grid type
* sycl : try to fix after IQ1_S changes * sycl : iq1s_grid -> iq1s_grid_gpu * sycl : fix grid type
| const int8_t * grid = (const int8_t *)(iq1s_grid + (x[i].qs[i8] | ((h & 8) << 5))); | ||
| const float d = (float)x[i].d * (2*(h & 7) + 1); | ||
| for (int j = 0; j < 8; ++j) y[j] = d * grid[j]; | ||
| const uint8_t * qs = x[i].qs + 8*ib; |
There was a problem hiding this comment.
This is wrong. There are no signs in IQ1_S and have never been. This bogus implementation has been sitting on the master branch for 2 weeks now. PR #6014 that actually fixes it, has been sitting unreviewed for 2 weeks.
There was a problem hiding this comment.
This current implementation does not have any effect on 1qs on our backend, and we are taking a look at the proposed solutions to identify a proper fix. I would not be so confident to say that it fixes it as with the changes mentioned here , I was not able to get the correct runs when compared with nv .
I would also request to avoid using strong language on public PRs if the intention is to collaborate, and in terms of the code it was an older version of your implementation which actually did not give any results on our backend and we are investigating a proper solution.
* sycl : try to fix after IQ1_S changes * sycl : iq1s_grid -> iq1s_grid_gpu * sycl : fix grid type
No description provided.