Eval bug: qwen35moe always forces a full prompt reprocess after each message, 'failed to truncate'

[timkhronos]

Name and Version

version: 8070 (cc45f2a)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 9950x3d + 5090

Models

The issue presents with several quants of
Qwen3.5 397B

Problem description & steps to reproduce

When a generation is initiated, regardless if from the llama.cpp webui, sillytavern or other frontends, every generation regardless if it's a new reply or a swipe forces a full prompt reprocessing.

launch arguments:

/llama-server -m model/Qwen3.5-397B-A17B-UD-Q5_K_XL-00001-of-00007.gguf -ngl 999 --threads 16 --threads-batch 16 --batch-size 2048 -ub 2048 -ot "blk.(0|1|2).ffn_.=CUDA0" -ot "blk.._exps.=CPU" --ctx-size 96000 --port 15000 --chat-template-kwargs "{"enable_thinking": false}" --mmproj Desktop/model/AI/LLama.cpp/mmproj-F32.gguf --no-mmap

First Bad Commit

It is present since the commit adding support for qwen3.5moe

Relevant log output

slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.838
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> logit-bias -> top-k -> min-p -> ?temp-ext -> adaptive-p
slot launch_slot_: id 3 | task 427 | processing task, is_child = 0
slot update_slots: id 3 | task 427 | new prompt, n_ctx_slot = 96000, n_keep = 0, task.n_tokens = 2189
slot update_slots: id 3 | task 427 | need to evaluate at least 1 token for each active slot (n_past = 2189, task.n_tokens() = 2189)
slot update_slots: id 3 | task 427 | n_past was set to 2188
slot update_slots: id 3 | task 427 | n_tokens = 2188, memory_seq_rm [2188, end)
slot update_slots: id 3 | task 427 | failed to truncate tokens with position >= 2188 - clearing the memory
slot prompt_clear: id 3 | task 427 | clearing prompt with 2188 tokens

[ebfio]

Can confirm, same thing happens to me.

llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | new prompt, n_ctx_slot = 125184, n_keep = 0, task.n_tokens = 37649
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | n_tokens = 36907, memory_seq_rm [36907, end)
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | failed to truncate tokens with position >= 36907 - clearing the memory
llamacpp-qwen-3.5  | slot prompt_clear: id  0 | task 4223 | clearing prompt with 36907 tokens
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.054397
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | n_tokens = 2048, memory_seq_rm [2048, end)
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.108794
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 4223 | n_tokens = 4096, memory_seq_rm [4096, end)```

[LifesLight]

Same thing happening to me, I also tested saving and restoring the KV cache which resulted in this output:

slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.997 (> 0.100 thold), f_keep = 0.998
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  0 | task 1 | processing task, is_child = 0
slot update_slots: id  0 | task 1 | new prompt, n_ctx_slot = 128000, n_keep = 0, task.n_tokens = 71240
slot update_slots: id  0 | task 1 | n_past = 71010, slot.prompt.tokens.size() = 71185, seq_id = 0, pos_min = 71184, n_swa = 1
slot update_slots: id  0 | task 1 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)

[ggerganov]

I can't run this model - do you see messages in the logs such as these:

2.00.966.566 W srv  get_availabl: updating prompt cache
2.00.969.724 W srv   prompt_save:  - saving prompt with length 5516, total state size = 80.001 MiB
2.00.996.568 W srv          load:  - looking for better prompt, base f_keep = 0.442, sim = 0.455
2.00.996.581 W srv        update:  - cache state: 2 prompts, 284.750 MiB (limits: 8192.000 MiB, 1048576 tokens, 1048576 est)
2.00.996.582 W srv        update:    - prompt 0xb3ace6490:    2432 tokens, checkpoints:  1,   109.513 MiB
2.00.996.583 W srv        update:    - prompt 0xb3ace6510:    5516 tokens, checkpoints:  2,   175.237 MiB
2.00.996.584 W srv  get_availabl: prompt cache update took 30.02 ms

[ebfio]

@ggerganov no, I don't. I only see:

llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | n_tokens = 57344, memory_seq_rm [57344, end)
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | prompt processing progress, n_tokens = 58182, batch.n_tokens = 838, progress = 0.991277
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | n_tokens = 58182, memory_seq_rm [58182, end)
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | prompt processing progress, n_tokens = 58694, batch.n_tokens = 512, progress = 1.000000
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | prompt done, n_tokens = 58694, batch.n_tokens = 512
llamacpp-qwen-3.5  | slot init_sampler: id  0 | task 10687 | init sampler, took 4.46 ms, tokens: text = 58694, total = 58694
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 10687 | created context checkpoint 3 of 8 (pos_min = 58181, pos_max = 58181, size = 186.329 MiB)

and

llamacpp-qwen-3.5  | slot update_slots: id  0 | task 9208 | prompt processing progress, n_tokens = 58160, batch.n_tokens = 512, progress = 1.000000
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 9208 | prompt done, n_tokens = 58160, batch.n_tokens = 512
llamacpp-qwen-3.5  | slot init_sampler: id  0 | task 9208 | init sampler, took 4.41 ms, tokens: text = 58160, total = 58160
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 9208 | erasing old context checkpoint (pos_min = 49427, pos_max = 49427, size = 186.329 MiB)
llamacpp-qwen-3.5  | slot update_slots: id  0 | task 9208 | created context checkpoint 8 of 8 (pos_min = 57647, pos_max = 57647, size = 186.329 MiB)

[timkhronos]

Same here. I only see:
First gen:

slot launch_slot_: id  3 | task -1 | sampler chain: logits -> pos_ban -> top-k -> min-p -> ?temp-ext -> adaptive-p 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 96000, n_keep = 0, task.n_tokens = 2524
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 2012, batch.n_tokens = 2012, progress = 0.797147
slot update_slots: id  3 | task 0 | n_tokens = 2012, memory_seq_rm [2012, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 2524, batch.n_tokens = 512, progress = 1.000000
slot update_slots: id  3 | task 0 | prompt done, n_tokens = 2524, batch.n_tokens = 512
slot init_sampler: id  3 | task 0 | init sampler, took 0.30 ms, tokens: text = 2524, total = 2524
slot update_slots: id  3 | task 0 | created context checkpoint 1 of 8 (pos_min = 2011, pos_max = 2011, size = 186.329 MiB)

regenerate:

slot launch_slot_: id  3 | task -1 | sampler chain: logits -> pos_ban -> top-k -> min-p -> ?temp-ext -> adaptive-p 
slot launch_slot_: id  3 | task 50 | processing task, is_child = 0
slot update_slots: id  3 | task 50 | new prompt, n_ctx_slot = 96000, n_keep = 0, task.n_tokens = 2524
slot update_slots: id  3 | task 50 | need to evaluate at least 1 token for each active slot (n_past = 2524, task.n_tokens() = 2524)
slot update_slots: id  3 | task 50 | n_past was set to 2523
slot update_slots: id  3 | task 50 | n_tokens = 2523, memory_seq_rm [2523, end)
slot update_slots: id  3 | task 50 | failed to truncate tokens with position >= 2523 - clearing the memory
slot prompt_clear: id  3 | task 50 | clearing prompt with 2523 tokens
slot update_slots: id  3 | task 50 | prompt processing progress, n_tokens = 2012, batch.n_tokens = 2012, progress = 0.797147
slot update_slots: id  3 | task 50 | n_tokens = 2012, memory_seq_rm [2012, end)
slot update_slots: id  3 | task 50 | prompt processing progress, n_tokens = 2524, batch.n_tokens = 512, progress = 1.000000
slot update_slots: id  3 | task 50 | prompt done, n_tokens = 2524, batch.n_tokens = 512
slot init_sampler: id  3 | task 50 | init sampler, took 0.32 ms, tokens: text = 2524, total = 2524

next message:

slot launch_slot_: id  3 | task -1 | sampler chain: logits -> pos_ban -> top-k -> min-p -> ?temp-ext -> adaptive-p 
slot launch_slot_: id  3 | task 97 | processing task, is_child = 0
slot update_slots: id  3 | task 97 | new prompt, n_ctx_slot = 96000, n_keep = 0, task.n_tokens = 2585
slot update_slots: id  3 | task 97 | n_tokens = 2520, memory_seq_rm [2520, end)
slot update_slots: id  3 | task 97 | failed to truncate tokens with position >= 2520 - clearing the memory
slot prompt_clear: id  3 | task 97 | clearing prompt with 2520 tokens
slot update_slots: id  3 | task 97 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.792263
slot update_slots: id  3 | task 97 | n_tokens = 2048, memory_seq_rm [2048, end)
slot update_slots: id  3 | task 97 | prompt processing progress, n_tokens = 2073, batch.n_tokens = 25, progress = 0.801934
slot update_slots: id  3 | task 97 | n_tokens = 2073, memory_seq_rm [2073, end)
slot update_slots: id  3 | task 97 | prompt processing progress, n_tokens = 2585, batch.n_tokens = 512, progress = 1.000000
slot update_slots: id  3 | task 97 | prompt done, n_tokens = 2585, batch.n_tokens = 512
slot init_sampler: id  3 | task 97 | init sampler, took 0.24 ms, tokens: text = 2585, total = 2585

[ebfio]

A bit more context in sequence:

llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | prompt processing progress, n_tokens = 32768, batch.n_tokens = 2048, progress = 0.958381
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | n_tokens = 32768, memory_seq_rm [32768, end)
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | prompt processing progress, n_tokens = 33679, batch.n_tokens = 911, progress = 0.985025
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | n_tokens = 33679, memory_seq_rm [33679, end)
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | prompt processing progress, n_tokens = 34191, batch.n_tokens = 512, progress = 1.000000
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | prompt done, n_tokens = 34191, batch.n_tokens = 512
llamacpp-qwen-3.5  | slot init_sampler: id  1 | task 2818 | init sampler, took 2.79 ms, tokens: text = 34191, total = 34191
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | erasing old context checkpoint (pos_min = 25643, pos_max = 25643, size = 186.329 MiB)
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 2818 | created context checkpoint 8 of 8 (pos_min = 33678, pos_max = 33678, size = 186.329 MiB)
llamacpp-qwen-3.5  | slot print_timing: id  1 | task 2818 |
llamacpp-qwen-3.5  | prompt eval time =   58658.51 ms / 34191 tokens (    1.72 ms per token,   582.88 tokens per second)
llamacpp-qwen-3.5  |        eval time =   21438.78 ms /  1002 tokens (   21.40 ms per token,    46.74 tokens per second)
llamacpp-qwen-3.5  |       total time =   80097.29 ms / 35193 tokens
llamacpp-qwen-3.5  | slot      release: id  1 | task 2818 | stop processing: n_tokens = 35192, truncated = 0
llamacpp-qwen-3.5  | srv  update_slots: all slots are idle
llamacpp-qwen-3.5  | srv  log_server_r: done request: POST /v1/chat/completions 10.0.0.100 200
llamacpp-qwen-3.5  | Template supports tool calls but does not natively describe tools. The fallback behaviour used may produce bad results, inspect prompt w/ --verbose & consider overriding the template.
llamacpp-qwen-3.5  | srv  params_from_: Chat format: Hermes 2 Pro
llamacpp-qwen-3.5  | slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.979 (> 0.100 thold), f_keep = 0.993
llamacpp-qwen-3.5  | slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist
llamacpp-qwen-3.5  | slot launch_slot_: id  1 | task 3843 | processing task, is_child = 0
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | new prompt, n_ctx_slot = 125184, n_keep = 0, task.n_tokens = 35717
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | n_tokens = 34959, memory_seq_rm [34959, end)
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | failed to truncate tokens with position >= 34959 - clearing the memory
llamacpp-qwen-3.5  | slot prompt_clear: id  1 | task 3843 | clearing prompt with 34959 tokens
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.057340
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | n_tokens = 2048, memory_seq_rm [2048, end)
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.114679
llamacpp-qwen-3.5  | slot update_slots: id  1 | task 3843 | n_tokens = 4096, memory_seq_rm [4096, end)

[ggerganov]

Oh I see now. It's because this model is multi-modal:

llama.cpp/tools/server/server-context.cpp

Lines 2284 to 2287 in afa6bfe

	// note: disallow with mtmd contexts for now
	// https://github.com/ggml-org/llama.cpp/issues/17043
	if (!mctx && n_past > 0 && n_past < slot.prompt.n_tokens()) {
	const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx), slot.id);

I've left a TODO to fix that:

llama.cpp/tools/server/server-context.cpp

Lines 2336 to 2338 in afa6bfe

	if (pos_min > pos_min_thold) {
	// TODO: support can be added in the future when corresponding vision models get released
	GGML_ASSERT(!slot.prompt.tokens.has_mtmd);

For now, you can disable the multi-modal part and it should be OK.

[LifesLight]

I am running the model without the mmproj and it still exhibits the same issue.
Server Command:

llama-server -b 4096 -ub 4096 -m "... Qwen3.5-397B-A17B-IQ4_XS.gguf" --slot-save-path "... Qwen3.5/" --threads 16 --cache-type-k f16 --cache-type-v f16 -c 64000 -ngl 999 --n-cpu-moe 999 -fa on --port "${PORT}" --kv-unified --parallel 1

Logs:

slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.159 (> 0.100 thold), f_keep = 0.172
srv  get_availabl: updating prompt cache
srv   prompt_save:  - saving prompt with length 192, total state size = 191.957 MiB
srv          load:  - looking for better prompt, base f_keep = 0.172, sim = 0.159
srv        update:  - cache state: 1 prompts, 191.957 MiB (limits: 8192.000 MiB, 64000 tokens, 64000 est)
srv        update:    - prompt 0x589c735f0340:     192 tokens, checkpoints:  0,   191.957 MiB
srv  get_availabl: prompt cache update took 116.61 ms
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  0 | task 157 | processing task, is_child = 0
slot update_slots: id  0 | task 157 | new prompt, n_ctx_slot = 64000, n_keep = 0, task.n_tokens = 208
slot update_slots: id  0 | task 157 | n_past = 33, slot.prompt.tokens.size() = 192, seq_id = 0, pos_min = 191, n_swa = 1
slot update_slots: id  0 | task 157 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 157 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 157 | prompt processing progress, n_tokens = 208, batch.n_tokens = 208, progress = 1.000000
slot update_slots: id  0 | task 157 | prompt done, n_tokens = 208, batch.n_tokens = 208
slot init_sampler: id  0 | task 157 | init sampler, took 0.02 ms, tokens: text = 208, total = 208

[timkhronos]

@ggerganov

I think the issue is that for some reason even during times when it shouldn't, the model wants to fall back on context-checkpoints, which are disabled when an mmproj is loaded.

So there is probably something model specific going weird here as well.

[ebfio]

@ggerganov

I think the issue is that for some reason even during times when it shouldn't, the model wants to fall back on context-checkpoints, which are disabled when an mmproj is loaded.

So there is probably something model specific going weird here as well.

I'm sure there is. I have another machine running Kimi K2.5 with the mmproj file and this doesn't happen.

[ggerganov]

@LifesLight This is normal - we don't do checkpoints for such short prompts (208 tokens). The prompt has to be at least 512 tokens to create the checkpoint.

I'm sure there is. I have another machine running Kimi K2.5 with the mmproj file and this doesn't happen.

Qwen 3.5 is the first hybrid multi-modal model. Kimi K2.5 is not hybrid - i.e. it does not have a recurrent state.

@timkhronos The checkpointing logic can be improved. There is an issue for that: #19298. But all the reports here so far are solved by simply disabling mmproj.

[ggerganov]

@timkhronos To clarify your specific case, since it is a bit different from the rest. Basically, you observe that it recomputes the prompt even when continuing the last message, which on first glance does not need checkpoints.

The issue is that the server logic currently needs to always process at least one token prior to entering the "generation phase". Hence the message:

need to evaluate at least 1 token for each active slot (n_past = 2189, task.n_tokens() = 2189)

Technically, you are correct. It's just the server logic currently does not support this. There might be a way to fix this - I'll give it some thought.

[LifesLight]

@ggerganov You're right, using a larger ctx chat fixed re-building the cache when adding a new message, but saving and restoring the kv-cache is definitely not working (not sure if this is a different issue or related, but might be relevant):
Using --no-mmap, chat saved -> server restarted -> cache restored -> message appended

slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.986 (> 0.100 thold), f_keep = 0.987
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  0 | task 1 | processing task, is_child = 0
slot update_slots: id  0 | task 1 | new prompt, n_ctx_slot = 48128, n_keep = 0, task.n_tokens = 18605
slot update_slots: id  0 | task 1 | n_past = 18342, slot.prompt.tokens.size() = 18591, seq_id = 0, pos_min = 18590, n_swa = 1
slot update_slots: id  0 | task 1 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 1 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 1 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 4096, progress = 0.220156

[pwilkin]

I was thinking of adding a configurable option of allowing the creation of a recurrent snapshot every n batches during prompt processing (currently it's hardcoded at a specific point before the end), I'll try to give it a go when I'm done with the autoparser.