CLBlast: q5_0, q5_1, q8_0 dequant kernels#1225
Conversation
|
For |
|
@ggerganov I did, that happens during the transfer to the GPU. It works just fine for Q5_1. Q5_0 has a different problem that I can only suspect is related to memory alignment. |
| cl_host_b = (cl_block_q5_0*) malloc(sizeof(cl_block_q5_0) * global / 32); | ||
| for (size_t i = 0; i < global / 32; i++) { | ||
| cl_host_b[i].d = ggml_fp16_to_fp32(b[i].d); | ||
| memcpy(&cl_host_b[i].qh, b[i].qh, sizeof(uint32_t) + QK5_0 / 2); |
There was a problem hiding this comment.
Use 2x memcpy - one for qh and one for qs
|
Btw might want to add a |
|
It was an alignment / padding issue. the In #1422 I added an |
* Implement q5_0, q5_1 and q8_0 * Work around q5_0 OpenCL issue * Fix q8_0 dequant kernel * Move cl kernels into ggml-opencl.c * Use two memcpy calls for q5_0 buffer transfer
* Implement q5_0, q5_1 and q8_0 * Work around q5_0 OpenCL issue * Fix q8_0 dequant kernel * Move cl kernels into ggml-opencl.c * Use two memcpy calls for q5_0 buffer transfer
* Implement q5_0, q5_1 and q8_0 * Work around q5_0 OpenCL issue * Fix q8_0 dequant kernel * Move cl kernels into ggml-opencl.c * Use two memcpy calls for q5_0 buffer transfer
4 participants
I had or still have an issue with q5_0 that I can't figure out. On Nvidia trying to transfer the quantized weights to the device leads to a CL_OUT_OF_RESOURCES error. On AMD and on POCL it leads to a segfault. It seems to have a problem with 22 byte structs, while 20 or 24 bytes are alright. I am not sure why this is the case.
As a workaround I copy the weights into a new struct and do the FP16 to FP32 conversion on CPU. This seems to have little overhead and works, but it should not be needed. If anyone knows what's up here please let me know.
I also moved the .cl file into the opencl.c as requested.