Eval bug: Infinite repetition loop in llama-server with peg-gemma4 parser during tool calls

[mamama1]

When using the newly implemented Gemma 4 (peg-gemma4) chat format in llama-server, the model enters an infinite repetition loop during tool-calling. The server appears to re-parse the entire model response turn for every single token generated, and the model continuously repeats the same tool call without ever reaching an EOT (End of Turn) or EOS.
Environment

Build: llama.cpp (full-vulkan)

WARNING: radv is not a conformant Vulkan implementation, testing use only.
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8643 (f49e917)
built with GNU 15.2.0 for Linux x86_64

Server: llama-server

Model: gemma-4-26B-A4B-it-UD-Q4_K_M.gguf

Actual Behavior

The model generates a tool call, but instead of finishing the turn, it repeats the same call indefinitely. The logs show the PEG parser running a full re-parse for every token:

The server continues to send streamed chunks for the same tool call index (e.g., index: 83) repeatedly. The model never exits the <|tool_call|> block.
Expected Behavior

The model should generate the tool call once, close the tags correctly (e.g., <tool_call|>), and stop generating to wait for the tool output.

Operating systems

Linux

GGML backends

Vulkan

Hardware

E5-2640v4 on Supermicro X10DRi with ReBAR BIOS Patch applied (32G BAR available)
Radeon AI Pro R9700 32G

Models

gemma-4-26B-A4B-it-UD-Q4_K_M.gguf

Problem description & steps to reproduce

Steps to Reproduce

Run llama-server with a Gemma 4 model using the default auto-detected peg-gemma4 format.
Provide a system prompt containing tool definitions (e.g., Home Assistant Assist tools).
Issue a prompt that triggers a tool call (e.g., "What is the outside temperature?").
Observe the server logs and the client response.

First Bad Commit

No response

Relevant log output

Plaintext

[llama-cpp-vulkan] Parsing PEG input with format peg-gemma4: <|turn>model
[llama-cpp-vulkan] <|channel>thought
[llama-cpp-vulkan] <channel|><|tool_call>call:assist__GetLiveContext{}...
[llama-cpp-vulkan] slot process_toke: id 0 | next token: 70940 'assist'

[llama-cpp-vulkan] Parsing PEG input with format peg-gemma4: <|turn>model
[llama-cpp-vulkan] <|channel>thought
[llama-cpp-vulkan] <channel|><|tool_call>call:assist__GetLiveContext{}...
[llama-cpp-vulkan] slot process_toke: id 0 | next token: 1269 '__'

[llama-cpp-vulkan] Parsing PEG input with format peg-gemma4: <|turn>model
...

[NickM-27]

I am seeing this on 7900XTX running vulkan as well, on the latest commit after the parser and tokenizer fixes

[dynamiccreator]

not just tool calls this happens during thinking as well. on lmstudio it works fine, but not llama.cpp server
linux, cuda rtx 3090. Build b8646

[NickM-27]

I always see these reasoning budget logs whenever the infinite output occurs, despite the fact that I have reasoning set to off

[50689] srv  get_availabl: prompt cache update took 877.71 ms
[50689] slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist 
[50689] slot launch_slot_: id  0 | task 241 | processing task, is_child = 0
[50689] slot update_slots: id  0 | task 241 | new prompt, n_ctx_slot = 20224, n_keep = 0, task.n_tokens = 8075
[50689] slot update_slots: id  0 | task 241 | n_past = 1195, slot.prompt.tokens.size() = 7853, seq_id = 0, pos_min = 5361, n_swa = 1024
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [5361, 7836] against 171...
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [5361, 7720] against 171...
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [5361, 7413] against 171...
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [2389, 5460] against 171...
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [1024, 4095] against 171...
[50689] slot update_slots: id  0 | task 241 | Checking checkpoint with [0, 2047] against 171...
[50689] slot update_slots: id  0 | task 241 | restored context checkpoint (pos_min = 0, pos_max = 2047, n_tokens = 2048, n_past = 1195, size = 212.524 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 0, pos_max = 2047, n_tokens = 2048, n_swa = 1024, pos_next = 1195, size = 212.524 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 1024, pos_max = 4095, n_tokens = 4096, n_swa = 1024, pos_next = 1195, size = 318.786 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 2389, pos_max = 5460, n_tokens = 5461, n_swa = 1024, pos_next = 1195, size = 318.786 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 5361, pos_max = 7413, n_tokens = 7414, n_swa = 1024, pos_next = 1195, size = 213.043 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 5361, pos_max = 7720, n_tokens = 7721, n_swa = 1024, pos_next = 1195, size = 244.901 MiB)
[50689] slot update_slots: id  0 | task 241 | erased invalidated context checkpoint (pos_min = 5361, pos_max = 7836, n_tokens = 7837, n_swa = 1024, pos_next = 1195, size = 256.938 MiB)
[50689] slot update_slots: id  0 | task 241 | n_tokens = 1195, memory_seq_rm [1195, end)
[50689] slot update_slots: id  0 | task 241 | prompt processing progress, n_tokens = 3243, batch.n_tokens = 2048, progress = 0.401610
[50689] slot update_slots: id  0 | task 241 | n_tokens = 3243, memory_seq_rm [3243, end)
[50689] slot update_slots: id  0 | task 241 | 2048 tokens since last checkpoint at 0, creating new checkpoint during processing at position 5291
[50689] slot update_slots: id  0 | task 241 | prompt processing progress, n_tokens = 5291, batch.n_tokens = 2048, progress = 0.655232
[50689] slot update_slots: id  0 | task 241 | created context checkpoint 1 of 32 (pos_min = 171, pos_max = 3242, n_tokens = 3243, size = 318.786 MiB)
[50689] slot update_slots: id  0 | task 241 | n_tokens = 5291, memory_seq_rm [5291, end)
[50689] slot update_slots: id  0 | task 241 | 2048 tokens since last checkpoint at 3243, creating new checkpoint during processing at position 6027
[50689] slot update_slots: id  0 | task 241 | prompt processing progress, n_tokens = 6027, batch.n_tokens = 736, progress = 0.746378
[50689] slot update_slots: id  0 | task 241 | created context checkpoint 2 of 32 (pos_min = 2219, pos_max = 5290, n_tokens = 5291, size = 318.786 MiB)
[50689] slot update_slots: id  0 | task 241 | n_tokens = 6027, memory_seq_rm [6027, end)
[50689] slot update_slots: id  0 | task 241 | prompt processing progress, n_tokens = 8071, batch.n_tokens = 2044, progress = 0.999505
[50689] slot update_slots: id  0 | task 241 | created context checkpoint 3 of 32 (pos_min = 2955, pos_max = 6026, n_tokens = 6027, size = 318.786 MiB)
[50689] slot update_slots: id  0 | task 241 | n_tokens = 8071, memory_seq_rm [8071, end)
[50689] reasoning-budget: activated, budget=2147483647 tokens
[50689] reasoning-budget: deactivated (natural end)
[50689] slot init_sampler: id  0 | task 241 | init sampler, took 0.59 ms, tokens: text = 8075, total = 8075
[50689] slot update_slots: id  0 | task 241 | prompt processing done, n_tokens = 8075, batch.n_tokens = 4
[50689] slot update_slots: id  0 | task 241 | created context checkpoint 4 of 32 (pos_min = 4999, pos_max = 8070, n_tokens = 8071, size = 318.786 MiB)
[50689] srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
srv  log_server_r: done request: POST /v1/chat/completions 192.168.50.105 200

[NickM-27]

Here is a verbose log file showing when this happens. It looks like it creates the tool call that I would expect (weather forecast) but then just continuously starts adding on tool calls after that.

Gemma4-repitition.txt

Been trying to debug with Claude, both just to see if this is something that I can temporarily fix on my side and also just for learning. This is what it says, I'd assume it might not be accurate but in any case I am going to try disabling parallel tool calling and see if that helps

The model is generating unlimited parallel tool calls and never stops. Looking at the token stream:

1. <|tool_call>call:weather-forecast__GetWeatherForecast{range:<|"|>week<|"|>}<tool_call|> -- valid call 1
2. <|tool_call>call:basic-utilities__calendar_day_info{day:4,...}<tool_call|> -- call 2 (April 4)
3. <|tool_call>call:basic-utilities__calendar_day_info{day:5,...}<tool_call|> -- call 3 (April 5)
4. ...day 6, 7, 8, 9, 10, 11, 12, 13... and keeps incrementing forever

The model decides it needs calendar info for each day of the week, but never stops -- it just keeps going day 4, 5, 6, 7, 8, 9, 10, 11, 12, 13...

Why nothing stops it

The grammar is lazy (because tool_choice=auto) and allows zero_or_more parallel tool calls
(chat-auto-parser-generator.cpp:715). Once triggered, the grammar happily accepts unlimited
<|tool_call>...<tool_call|> sequences. And n_remaining = -1 means no max_tokens limit is set.

Workarounds

For now, you can mitigate this with any of:

1. "parallel_tool_calls": false in the request body -- forces single tool call per response
2. Set max_tokens to a reasonable limit (e.g. 1024) to cap runaway generation
3. "tool_choice": "required" makes the grammar non-lazy, which would constrain to known tool
   names only (though still allows repetition)

The underlying issue is that the Gemma4 model gets into a repetitive parallel tool call pattern and
the grammar has no upper bound. This is arguably a bug in the parser generator -- zero_or_more
should probably be repeat(0, N) with some reasonable max. Worth filing as an issue on the
llama.cpp repo.

[NickM-27]

It does appear that parallel tool calling disabled and it has not had the same behavior.

[aldehir]

Make sure you're at least on b069b10, there was an issue with the tokenizer.

[50689] reasoning-budget: activated, budget=2147483647 tokens
[50689] reasoning-budget: deactivated (natural end)

A budget of INT_MAX is effectively off. The reasoning sampler is only counting, it doesn't mask any logits in this mode. Still, the logs are confusing and we can suppress them.

[NickM-27]

I am currently running d006858 as I have my own fork which builds vulkan server automatically.

I continue to not have this issue after disabling parallel tool calling. Happy to provide any info that is helpful

[mamama1]

as soon as a new docker container is available, i can test

[aldehir]

Give #21418 a try for the parallel tool calling issues, if you can.

[mamama1]

because i'm stupid, i can only easily test whats in the most recent container release.certainly i can confirm though that the tool calling infinite loop is indeed stopped by disabling parallel tool calling as of build 8660 in my case.

[NickM-27]

@aldehir Have been testing for about 30 minutes and everything seems to be working great for me

[xhedit]

@aldehir #21418 fixed this issue for me

[wbste]

Having same issue.

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
load_backend: loaded CUDA backend from C:\llama\ggml-cuda.dll
load_backend: loaded RPC backend from C:\llama\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llama\ggml-cpu-alderlake.dll
version: 8708 (ae65fbdf3)
built with Clang 19.1.5 for Windows x86_64

Downloaded the most recent version released a few hours ago from unsloth.

About 3 out of 4 times I can trigger it in the llama.cpp web ui.

can you get the forecast and time?

It then calls my mcp server that I've used forever with the same tools, and it'll loop/add digits? Does seem to be parallel tool call related. I've tried multiple templates...

• built in.
• Patching the built in with the proposed from transformers.
• The interleaved one in the llama.cpp repo.
• No noticeable difference in output.

The docs say parallel is disabled by default but maybe that's a bit outdated?

rg "I'll call both"
llama-server-service_20260408.out.log
22865:[52070] Both functions seem relevant and can fulfill the request. I'll call both
22873:[52070] Both functions seem relevant and can fulfill the request. I'll call both
22899:[52070] Both functions seem relevant and can fulfill the request. I'll call both.
22907:[52070] Both functions seem relevant and can fulfill the request. I'll call both.
22933:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|>
22941:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|>
22963:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>
22971:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>
22993:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call
23001:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call
23023:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:
23031:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:
23053:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get
23061:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get
23083:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_
23091:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_
23113:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast
23121:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast
23143:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}
23151:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}
23176:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|>
23184:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|>
23206:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>
23214:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>
23236:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call
23244:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call
23266:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:
23274:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:
23296:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:6
23304:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:6
23326:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:60
23334:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:60
23356:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:604
23364:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:604
23386:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:6047
23394:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:6047
23416:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:60471
23424:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:60471
23446:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:604713
23454:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:604713
23476:[52070] Both functions seem relevant and can fulfill the request. I'll call both.<channel|><|tool_call>call:get_forecast{}<tool_call|><|tool_call>call:6047135

[aldehir]

The WebUI might not be sending back the appropriate info. I expect the name the tool response to be the tool name and not the tool call id...

Is this only under the webui?