SYCL backend error PI_ERROR_INVALID_WORK_GROUP_SIZE on iGPU UHD 770

[fakezeta]

When offloading to iGPU UHD 770 in a docker from https://github.com/mudler/LocalAI (b2128) llama.cpp crashes with the following error:

The number of work-items in each dimension of a work-group cannot exceed {512, 512, 512} for this device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp, line:12708

From trial and error it happens if I have number of tokens predicted >256. I mean that if I limit the tokens with 256 it does not happen.

Tested with multiple 7b mistral models with both Q6 and Q8 quantization

Intel oneAPI version 2024.0

[airMeng]

Hi @fakezeta UHD770 might be limited nether on performance or functionality, as issued in the README

[fakezeta]

Thank you @airMeng, I read the README but it's only giving warning on performance not functionality.
Tested with OpenVINO, the same oneAPI version and the same models I had not experienced any issues (and 6x performance but this is another story :) )

[NeoZhangJianyu]

@fakezeta
I guess https://github.com/mudler/LocalAI use very old ggml for SYCL library.
This issue has been fixed in llama.cpp.

Maybe you could try with llama.cpp on iGPU UHD 770, refer to sycl guide
If it's OK, that approve my guess.
You need to ask LocalAI to upgrade ggml for SYCL.

[fakezeta]

Sorry for the late reply. Tested with a more recent build and I can confirm that it's working fine.

Thank you.

[CarlGao4]

I'm using b5216 to run Qwen3MoE models, and the error still exists. I've found another issue: intel/ipex-llm#12839

It seems that it is something related to MoE models? Besides, I've found that it can be successfully deployed if I only load 1 layer to GPU. I'm also sure that even loading the whole model to GPU would not cause out of memory

[Columpio]

@CarlGao4
Qwen3MoE works for me with the following params: llama-cli.exe -m D:\llmstudio_models\unsloth\Qwen3-30B-A3B-GGUF\Qwen3-30B-A3B-IQ4_XS.gguf -ngl 49 -b 50 -ub 30 -f .\prompt.txt -sm none -s 0 -t 16 -c 4096 --keep 79 --repeat-penalty 1.1 --frequency-penalty 0.0 --top-k 20 --top-p 0.95 --min-p 0.0 --temp 0.6 -no-cnv --prio 3 -ot ".ffn_.*_exps.=CPU" -sys "/no_think" --override-kv "qwen3moe.expert_used_count=int:8" --mlock

The most important guys here are -b and -ub. If I increase -ub to 35, I get the same The number of work-items in each dimension of a work-group cannot exceed {512, 512, 512} for this device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp, line:12708 . So my general advice is to take my above params and reduce -ub if you get this error.

P.S., on my Intel Iris Xe with SYCL I still get only:

llama_perf_context_print: prompt eval time =    2682.75 ms /    67 tokens (   40.04 ms per token,    24.97 tokens per second)
llama_perf_context_print:        eval time =   43467.20 ms /   183 runs   (  237.53 ms per token,     4.21 tokens per second)

The fastest among all Windows backends on my GPU-poor (Intel Xe only) machine is OpenBLAS:

llama_perf_context_print: prompt eval time =    2231.52 ms /    67 tokens (   33.31 ms per token,    30.02 tokens per second)
llama_perf_context_print:        eval time =   39124.36 ms /   475 runs   (   82.37 ms per token,    12.14 tokens per second)

(with: .\llama-cli.exe -m D:\llmstudio_models\unsloth\Qwen3-30B-A3B-GGUF\Qwen3-30B-A3B-IQ4_XS.gguf -b 50 -ub 30 -f ..\llama\prompt.txt -s 0 -t 16 -c 4096 --keep 79 --repeat-penalty 1.1 --frequency-penalty 0.0 --top-k 20 --top-p 0.95 --min-p 0.0 --temp 0.6 -no-cnv --prio 3 -ot ".ffn_.*_exps.=CPU" -sys "/no_think" --override-kv "qwen3moe.expert_used_count=int:8" --mlock)

[CarlGao4]

@Columpio Which version are you using? I've even reduced ub to 8 and b to 16 on version b5216 and still getting this error

I'm also using Iris Xe GPU as my Nvidia GPU do not have enough VRAM

[Columpio]

@CarlGao4 I downloaded llama-b5219-bin-win-sycl-x64.zip from https://github.com/ggml-org/llama.cpp/releases/tag/b5219

[Columpio]

@CarlGao4 probably RAM really matters?

(I used dxdiag to view it)

[Columpio]

@CarlGao4 I installed everything as follows:

1. install https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html
2. cmd run: "C:\Program Files (x86)\Intel\oneAPI\setvars.bat\" intel64
3. copy, extract and cd to https://github.com/ggml-org/llama.cpp/releases/download/b5219/llama-b5219-bin-win-sycl-x64.zip
4. run llama-cli.exe -m D:\llmstudio_models\unsloth\Qwen3-30B-A3B-GGUF\Qwen3-30B-A3B-IQ4_XS.gguf -ngl 49 -b 50 -ub 30 -f .\prompt.txt -sm none -s 0 -t 16 -c 4096 --keep 79 --repeat-penalty 1.1 --frequency-penalty 0.0 --top-k 20 --top-p 0.95 --min-p 0.0 --temp 0.6 -no-cnv --prio 3 -ot ".ffn_.*_exps.=CPU" -sys "/no_think" --override-kv "qwen3moe.expert_used_count=int:8" --mlock

[CarlGao4]

I'm using i9-13900H with 64GiB DDR5 so I'm sure that my memory is efficient.
Besides, I'v compiled PyTorch with ipex support before, so I've Intel oneAPI installed correctly.
However, my error message does not look like yours

The number of work-items in each dimension of a work-group cannot exceed {512, 512, 512} for this deviceException caught at file:D:\a\llama.cpp\llama.cpp\ggml\src\ggml-sycl\ggml-sycl.cpp, line:3220

Your error occurs on line 12708 but mine is 3220

[NeoZhangJianyu]

@CarlGao4
The issue above is due to the your iGPU can't support big work-group (>512,512,512).

1. What's the llama.cpp you use?
   ggml-org/llama.cpp or IPEX-LLM.
   We only support the open source llama.cpp (ggml-org/llama.cpp).
   IPEX-LLM is close source. If you are using it, please ask them for support.

2. What's the model name and cmd?
   Can you share the whole log？