[BUG] Gemma 4 token repetition collapse during long generation — affects both 31b Dense and 26b MoE

[rnh0]

Summary

Both gemma-4-31B-it (Dense) and gemma-4-26B-A4B-it (MoE) exhibit token-level repetition collapse during long text generation. A word or token fragment doubles, then collapses into a single repeated token that fills the remaining generation budget. This occurs most reliably when combined with grammar-constrained (structured JSON) output, but the underlying tendency toward word doubling is visible even in unconstrained generation.

This may be related to #610, which reports repetition loops in the 26B MoE model during list generation. Our findings extend the scope to the 31b Dense variant and provide systematic isolation of the trigger conditions.

Evidence

Token doubling in unconstrained output

Even without any grammar constraint, successful outputs from both models show word doubling:

• "the act of observing a a peaceful environment"
• "the the waves", "sapphire sapphire"

This suggests the model's logit distribution has an inherent tendency to degenerate toward repeating the previous token, which does not self-correct reliably during sustained generation.

Repetition collapse with grammar-constrained output

When using structured JSON output (Ollama's format= parameter, which uses llama.cpp's grammar sampling), the word doubling escalates into a full collapse:

1. Normal generation starts fine inside a JSON string value
2. A word doubles: "contemplative contemplative"
3. Collapses into a single repeated token: "own own own own own..."
4. Fills remaining generation budget (thousands of tokens)

The grammar constraint cannot prevent this because repeated words are valid JSON string content.

Test results (Ollama, 10 seeds per test)

Test	gemma4:26b (MoE)	gemma4:31b (Dense)
	Repetition / Valid JSON	Repetition / Valid JSON
Short output + JSON schema	4/10 / 1/10	7/10 / 1/10
1000+ words + JSON schema	5/10 / 1/10	10/10 / 0/10
Complex nested schema	4/10 / 0/10	7/8 / 1/8
6 free-text fields	4/10 / 0/10	3/10 / 7/10
No JSON schema (free generation)	0/10 / n/a	0/10 / n/a

Key observations:

• Both architectures are affected — Dense (31b) has higher repetition rates, but MoE (26b) is equally broken for JSON validity
• Without grammar constraints, no repetition loops occur — but the word-doubling precursor is still present
• gemma3:27b is clean (0/10 repetition, 10/10 valid JSON on the same tests) — this is a gemma4-generation regression
• repeat_penalty has no effect — tested at 1.0, 1.15, and 1.5, identical seeds fail identically at all values
• Repeated tokens differ between architectures: 31b produces clean English words ("own", "beach", "same"), 26b produces more exotic fragments ("$\text{}$", "visually-cent,", "sing_er,")

Minimal reproduction

Requires Ollama with gemma4:31b or gemma4:26b. Text-only, no images needed.

import ollama

SCHEMA = {
    "type": "object",
    "required": ["description", "analysis", "tags"],
    "properties": {
        "description": {
            "type": "string",
            "description": "At least 3 detailed sentences.",
        },
        "analysis": {
            "type": "string",
            "description": "Several paragraphs of analysis.",
        },
        "tags": {"type": "array", "items": {"type": "string"}},
    },
}

response = ollama.chat(
    model="gemma4:31b",  # also reproduces with gemma4:26b
    messages=[{"role": "user", "content": (
        "Describe a beach scene at sunset in detail. "
        "Write at least 3 full sentences for description "
        "and several paragraphs for analysis."
    )}],
    format=SCHEMA,
    options={
        "num_ctx": 32768,
        "num_predict": 8192,
        "repeat_penalty": 1.15,
        "repeat_last_n": 256,
        "seed": 0,
    },
)
print(response.message.content[-200:])

Environment

• Ollama 0.20.5 (llama.cpp backend)
• NVIDIA RTX 5090 (32 GB VRAM)
• Ubuntu 24.04.3 LTS
• Also reported on Cloudflare Workers AI ([BUG] Gemma-4-26B-A4B: Deterministic Repetition Loop at 14th item in list ("Wait, I found it. The 14.") #610 comment) and LMStudio ([BUG] Gemma-4-26B-A4B: Deterministic Repetition Loop at 14th item in list ("Wait, I found it. The 14.") #610), suggesting this is not backend-specific

Related issues

• [BUG] Gemma-4-26B-A4B: Deterministic Repetition Loop at 14th item in list ("Wait, I found it. The 14.") #610 — Gemma 4 26B repetition loop during list generation
• gemma4:31b repetition loop during constrained JSON generation with free-text string fields ollama/ollama#15502 — Detailed isolation and stress testing of this bug
• Eval bug: Gemma 4 generates <unused24> tokens ggml-org/llama.cpp#21321 — Different bug: <unused24> tokens from GEMV buffer overlap (distinct root cause, fixed in b8702)

Why this is likely a model-level issue

1. Both architectures (Dense 31b and MoE 26b) are affected
2. gemma3:27b on the same inference engine with the same parameters is clean
3. The word-doubling precursor appears even without grammar constraints
4. The bug reproduces across multiple backends (Ollama, LMStudio, Cloudflare — per [BUG] Gemma-4-26B-A4B: Deterministic Repetition Loop at 14th item in list ("Wait, I found it. The 14.") #610)
5. Grammar-constrained sampling merely exposes and amplifies an underlying model tendency

---

Tested on 2026-04-11. Full test data and scripts: ollama/ollama#15502.

The test methodology was designed with assistance from Claude Code (Anthropic). All tests were run locally. Results are deterministic and independently reproducible.

[T-Klug]

Also occurs on omlx. Exact same behavior repetition penalty also did not help. Both thinking and no thinking produce the same behavior.

[oleg-savko]

same with me with vllm (and after stuck model not response while not restart it)

do someone find workaround how to prevent/fix it?
and is it make sense and watch that it can be fixed, or after release model it can be changed and fixed..?

[KarolKski]

Not yet. Our use case is translation only (HTML product descriptions).

Tried multiple quants (RedHatAI NVFP4, cyankiwi AWQ-4bit), multiple chat templates, and both vLLM 0.19 and nightly + transformers 5.5.0, all show the same failure mode for us. Model breaks on long generations when combined with structuredoutput:(json) and starts degenerating: repeated token runs, NBSP spam inside tag attributes, hallucinated class="text-xl" / idea="…" attributes, unbalanced HTML, eventually hitting finish_reason=length / max_tokens.

Adjusting sampling (tried Gemma-recommended top_p=0.95, top_k=64, and temperature variations including 0.7) did not change behavior. Same prompt template on Gemma-3 27B QAT runs clean at ~99.7% , so we rolled back to gemma 3.

[pdevine]

@rnh0 I think your agent meant "Ollama 0.20.5 (GGML backend)". There is no router in Ollama to llama.cpp for that model, only the GGML backend. I'm curious if llama.cpp also is suffering from the same problem though.

[curreta]

@rnh0 I think your agent meant "Ollama 0.20.5 (GGML backend)". There is no router in Ollama to llama.cpp for that model, only the GGML backend. I'm curious if llama.cpp also is suffering from the same problem though.

Last night I was running into this issue with Ollama, tested up to 0.20.8 and it still occurred. This morning I switched all my local inference to llama-swap (a light router layer over llama.cpp) and the bug no longer occurs. There are other repeating issues I'm coming across but the single/few token endless repeat is gone. I'll update as I keep tackling but so far I'm considering it addressed for my needs by using llama.cpp.

edit: Just had first trigger of it while using it in opencode. Looks like llama.cpp does not entirely solve it.

[rnh0]

@pdevine:
I think your agent meant "Ollama 0.20.5 (GGML backend)". There is no router in Ollama to llama.cpp for that model, only the GGML backend. I'm curious if llama.cpp also is suffering from the same problem though.

Good catch — you're right. Ollama uses its new GGML-based engine for Gemma 4, not llama.cpp. Apologies for the misattribution.

On your question about llama.cpp: it does not reproduce. I ran the same prompt + JSON schema that fails on Ollama, against llama.cpp:server-cuda with the same GGUF file (unsloth/gemma-4-26B-A4B-it-GGUF UD-Q4_K_XL), 10 seeds each:

• llama.cpp-server + MoE 26B GGUF + response_format=json_schema: 0/10 rep, 10/10 valid JSON
• llama.cpp-server + Dense 31B GGUF + response_format=json_schema: 0/10 rep, 10/10 valid JSON

Same weights, same prompt, same schema — clean on llama.cpp, 5-10/10 failure on Ollama.

I also ran the same pattern on vLLM (with AWQ-4bit since MoE GGUF loading still depends on huggingface/transformers#45296). With vLLM's default xgrammar backend the bug reproduces but as a whitespace-pad degenerate loop rather than word-phrase repetition — same structural failure (low-entropy trap inside the grammar's allowed language), different token. Setting StructuredOutputsConfig(disable_any_whitespace=True) flips it from 0/10 valid to 9/10 valid.

That suggests the underlying cause is how the structured-output/grammar constraint interacts with the model, not the model weights themselves. Full matrix and vLLM-side workaround: vllm-project/vllm#40080 (comment linked there).

I'd guess Ollama's new GGML engine has a similar whitespace-permissive grammar path, which is why the 5-10/10 Ollama rate aligns with the ~0/10 vLLM rate until xgrammar's whitespace handling is tightened. Might be worth checking whether Ollama's grammar sampler has an analogous knob. Happy to close this if the Gemma maintainers agree the picture is now "downstream structured-output path" rather than "Gemma-side concern."

---

The matrix above was designed with assistance from Claude Code (Anthropic). All tests were run locally and are deterministic. Repro scripts + raw outputs: https://gist.github.com/rnh0/e02a668c875af46eb5cb46ab0c77132b

[dotmobo]

With vLLM, in many of the responses, the word 'own' appears randomly, even in answers written entirely in French.

[TheAlex25]

Words sometimes are misspelled with Gemma-4-26B-A4B, and repeat glitches rarely happen. Would be nice if the settings on LM Studio could be better to prevent this. Great model nonetheless. Avoids sycophancy compared to older models. Best local LLM I have ever run.

Settings
Temp: 0.65
TopP = 0.95
MinP = 0.03
TopK = 40
Repetition Penalty: 1.1

LM Studio Version 0.4.10
Gemma-4-26B-A4B Q4_K_M (LM Studio Community version)
Ubuntu 25.10, Intel Core Ultra 9 185H, 4050 GPU
10 threads activated on CPU. 2 layers of model on GPU