Eval bug: Gemma 3 on ARM64 CPU (Cortex-A76) produces wrong logits; Qwen unaffected — b8816

[mhamann]

Name and Version

version: b8816 (also reproduced on b8708)
built: ubuntu-24.04-arm runner, cmake -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a
host: Raspberry Pi 5 (Cortex-A76, ARMv8.2-A, features: fphp asimdhp asimddp asimdrdm)

Operating systems

Linux

GGML backends

CPU

Hardware

Raspberry Pi 5 (Broadcom BCM2712), 4× Cortex-A76 @ 2.4 GHz, 8 GB RAM. ARMv8.2-A with half-precision + dot product. No SVE.

Models

Primary repro: Gemma 3 270M fine-tune (base: unsloth/functiongemma-270m-it), quantized to Q8_0.

Also reproduced to a lesser degree on: unsloth/gemma-3-1b-it-GGUF (Q8_0), which segfaults on b8708 and produces numerically wrong (but non-crashing) output on b8816.

Problem description & steps to reproduce

Same GGUF on Mac CPU produces correct output; on the Pi ARM64 CPU build the forward pass silently produces a wrong probability distribution.

Reproduction via llama-cpp-python API (or llama-cli directly — the result is the same):

from llama_cpp import Llama
llm = Llama(model_path="/path/model.Q8_0.gguf", n_ctx=512, n_gpu_layers=0, logits_all=True)
out = llm.create_completion("The capital of France is", max_tokens=1, temperature=0, logprobs=10)

Mac CPU (llama-cpp-python 0.3.20, bundled ggml 0.9.11):

top tokens:
  ' Paris': -0.003
  ' **':    -6.139
  ' Bast':  -8.627
  ...

Pi ARM64 CPU (b8816 built from GitHub workflow with -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a, ggml 0.9.11):

top tokens:
  ' bel':   -0.602
  ' el':    -1.468
  ' where': -2.533
  ' bal':   -2.695
  ' the':   -3.475
  ...  (" Paris" not in top 10)

Identical garbage with quantization = Q8_0 and bf16 (different garbage between quants, but still no "Paris" on Pi). Identical across n_threads=1/2/4. Same behavior with flash_attn on/off.

Isolated the bug to Gemma 3 architecture specifically:

• Qwen 2.5 0.5B Q8_0 on same Pi/binary: " Paris" at -0.918 ✓ (matches Mac to within expected precision)
• Gemma 3 1B base Q8_0: segfaults on b8708; produces wrong-but-not-crashing output on b8816 (" France" top instead of " Paris")
• Our 270M Gemma 3 fine-tune: byte-identical garbage logits between b8708 and b8816

The binary strings show standard Gemma 3 ISWA implementation symbols (llm_build_gemma3, llama_kv_cache_iswa). Qwen uses standard GQA and is unaffected. This strongly points at the ARM64 CPU kernel path for Gemma 3's interleaved sliding window attention or an upstream fp16 accumulation issue specific to this architecture/shape.

First Bad Commit

Bisection not performed yet — b8708 and b8816 both show the symptom. The 1B segfault does get fixed somewhere in that range, but the 270M numerics bug is present across the full range tested.

Relevant log output

llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 2048
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_kv_cache_iswa: using full-size SWA cache
llama_kv_cache_iswa: creating non-SWA KV cache, size = 4096 cells
llama_kv_cache_iswa: creating     SWA KV cache, size = 4096 cells
llama_kv_cache: K (f16): 6.00 MiB, V (f16): 6.00 MiB
llama_kv_cache: K (f16): 30.00 MiB, V (f16): 30.00 MiB
sched_reserve:        CPU compute buffer size =  2058.01 MiB
Warmup completed in 49.2ms

No errors at load time. The model loads cleanly, reports the expected graph layout, and warmup completes. The divergence appears in the forward pass logits directly.

I'm happy to bisect the range and try additional reproductions if useful. Also happy to share the fine-tuned GGUFs privately if helpful.

[mhamann]

Bisection update — regression confirmed between b7694 and b8708.

Built llama-cli from source for both tags on the same Pi 5 / Cortex-A76 / Debian 12 environment. Same GGUF, same prompt ("The capital of France is"), temperature=0:

• b7694 (2026-01-10): correct → "Paris"
• b8708 (2026-04-08): garbage → starts with "bel", never produces "Paris"
• b8816 (2026-04-16): garbage → byte-identical logits to b8708

Qwen 2.5 0.5B produces correct output on all three builds from the same host.

So the regression isn't the Gemma 4 churn alone — something between 2026-01-10 and 2026-04-08 broke Gemma 3 text inference on ARM64 CPU specifically. That's a ~1000 commit window. The graph-structural refactor in #21612 (gemma: perform per-layer projections in the first layer) is a plausible suspect since it moved per-layer ops out of the main layer loop, but I haven't git-bisected yet.

Happy to bisect the range if it'd help; let me know if there's a preferred approach (ARM runner available?) since each cross-build cycle is ~60s on my end.

[mhamann]

Update: this is NOT a source-code regression — it's a toolchain/build regression.

I built llama.cpp at b8708 (the tag my distro shipped) in two different containers and tested the resulting binary on the same Pi 5 / Cortex-A76 / Debian 12 bookworm target:

Build container	Compiler	Result on Pi
ubuntu:24.04 arm64 (matches llama-farm CI)	GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)	✗ garbage ("bel…")
debian:12 arm64	GCC 12.2.0 (Debian 12.2.0-14+deb12u1)	✓ correct ("Paris")

Same source tag (b8708), same cmake flags (-DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a -DBUILD_SHARED_LIBS=ON), same GGUF, same test host, same llama-cli arguments. The only material difference is the compiler.

Qwen 2.5 produces "Paris" correctly with both builds — so the miscompile is architecture-specific within the llama.cpp codebase (Gemma 3 interleaved-SWA codepath is the prime suspect), and likely gets exposed by something the GCC 13 optimizer does that GCC 12 doesn't. I git-bisected the source range b7694..b8708 assuming a code regression, and every source commit I tested passed — consistent with "source is fine, compiler miscompiles one of its kernels."

This takes the fix out of llama.cpp's hands and into the build configuration of whoever ships the ARM64 binary. For llama-farm specifically, a one-line change to pin the build container to debian:12 (or install gcc-12 on ubuntu-24.04-arm and set CC/CXX) resolves the garbage output.

Happy to close this issue, but leaving it open since this looks like a real GCC 13 autovectorization miscompile of one of ggml-cpu's ARM kernels that someone with more time might want to reduce. If useful, the exact reproducer:

docker build --platform=linux/arm64 -t repro - <<'DF'
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y build-essential cmake git
WORKDIR /src
RUN git clone https://github.com/ggml-org/llama.cpp.git . && git checkout b8708
RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
    -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_CURL=OFF \
    -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8-a \
 && cmake --build build -j --target llama-cli
DF

Run llama-cli --model <gemma-3-270M-Q8_0>.gguf --prompt "The capital of France is" --predict 5 --temp 0 -no-cnv on a Cortex-A76; then rebuild with FROM debian:12 and observe the same invocation now succeeds.

[mhamann]

Retraction: this was never a llama.cpp bug.

Re-ran the same reproduction against the unmodified Ubuntu 24.04 + GCC 13.3 build after fixing two bugs in my Python bindings layer on the consumer side, and the garbage output disappeared entirely. The binary you ship is fine; the symptoms I reported were caused by two problems in llamafarm-llama (my project's wrapper) that interacted with — not were caused by — llama.cpp:

1. _apply_chat_template was calling llama_chat_apply_template(NULL, ...), relying on an older behavior where NULL meant "use the model's template". Since b7376 that silently falls back to ChatML. Every non-ChatML model was getting its messages wrapped in <|im_start|>/<|im_end|>. Fix on our side was to call llama_model_chat_template(model, NULL) first and pass the returned string.

2. create_completion tokenized raw prompts with add_special=False, stripping BOS. Decoder-only models (Gemma in particular) degenerate badly without BOS — top-token logprob shifted from " Paris" -0.003 to " bel" -0.602. Fix on our side was add_special=True.

With both fixed, the GCC 13.3 binary produces identical output to the GCC 12 binary end-to-end. The GCC 12 workaround I described was a red herring — my bisect method conflated llama-cli (which tokenizes with BOS correctly) vs the Python wrapper (which didn't), so I misattributed the divergence to the only obvious variable I saw changing, which was the compiler.

Closing; no action needed on llama.cpp's side. Thanks, and sorry for the noise.