Eval bug: gemma-4-26b-a4b-it-turbo using tbq4_0 is insane (and gemma3 fails to load correctly)

[zekrom-vale]

Name and Version

zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --remove-orphans --entrypoint llama-cli turboa-llama-cpp   --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf   --mmproj /models/mmproj-google_gemma-4-26B-A4B-it-f16.gguf   --flash-attn on   --mlock   --no-warmup   --parallel 4   --n-gpu-layers -1   --n-cpu-moe 18   --cache-type-k tbq4_0   --cache-type-v tbq4_0   --threads 12   --ctx-size 128000   --batch-size 4096   --ubatch-size 1024   --n-predict 4096   -p "Why is the sky blue?" --version
Container turboquantamesianx-turboa-llama-cpp-run-7a90e7ab92d8 Creating 
Container turboquantamesianx-turboa-llama-cpp-run-7a90e7ab92d8 Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB
version: 8702 (cf2170e5f)
built with GNU 13.3.0 for Linux x86_64

The 1.5.2 version

Operating systems

Docker compose

Builder: nvidia/cuda:13.1.1-devel-ubuntu24.04
Runtime: nvidia/cuda:13.1.1-runtime-ubuntu24.04

docker-compose.yml

Dockerfile.txt

GGML backends

CUDA

Hardware

Ryzen 9900x + 5070Ti @PCIe5 + 64GB DDR5

Models

gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF
gemma-3-27b-it-IQ4_XS.gguf https://huggingface.co/unsloth/gemma-3-27b-it-GGUF

Problem description & steps to reproduce

When I set --cache-type-k tbq4_0 --cache-type-v tbq4_0 for Gemma4 26B A4B Q6_K_XL it produces gibberish just like it did with TheTom's version with turbo4. Even with out mmproj.

docker compose run --rm --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --mmproj /models/mmproj-google_gemma-4-26B-A4B-it-f16.gguf \
  --flash-attn on \
  --mlock \
  --no-warmup \
  --parallel 4 \
  --n-gpu-layers -1 \
  --n-cpu-moe 18 \
  --cache-type-k tbq4_0 \
  --cache-type-v tbq4_0 \
  --threads 12 \
  --ctx-size 128000 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --n-predict 4096 \
  -p "Why is the sky blue?"

Note, I tried to run Gemma3 and it failed to init at all, no errors just stopped. It works fine with q4_0:

zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --remove-orphans --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-3-27b-it-IQ4_XS.gguf \
  --fit off \
  --flash-attn on \
  --mlock \
  --no-warmup \
  --parallel 1 \
  --n-gpu-layers 50 \
  --cache-type-k tbq3_0 \
  --cache-type-v tbq3_0 \
  --threads 12 \
  --ctx-size 16000 \
  --batch-size 2048 \
  --ubatch-size 512 \
  -p "Why is the sky blue?"
Container turboquantamesianx-turboa-llama-cpp-run-f4ce0c6cce1b Creating 
Container turboquantamesianx-turboa-llama-cpp-run-f4ce0c6cce1b Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model... /

First Bad Commit

I am not sure, though it did appear not to be stable from the 1.5.0 version.

Relevant log output

Logs

<!-- Copy-pasted short logs go into the "console" area here -->
zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --remove-orphans --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --mmproj /models/mmproj-google_gemma-4-26B-A4B-it-f16.gguf \
  --flash-attn on \
  --mlock \
  --no-warmup \
  --parallel 4 \
  --n-gpu-layers -1 \
  --n-cpu-moe 18 \
  --cache-type-k tbq4_0 \
  --cache-type-v tbq4_0 \
  --threads 12 \
  --ctx-size 128000 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --n-predict 4096 \
  -p "Why is the sky blue?"
Container turboquantamesianx-turboa-llama-cpp-run-5f31db41abcb Creating 
Container turboquantamesianx-turboa-llama-cpp-run-5f31db41abcb Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8702-cf2170e5f
model      : gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
modalities : text, vision

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern
  /image <file>       add an image file


> Why is the sky blue?

瑞 de- actually-----—--—–--------—-
10.-10-1-1-1--1-1--1-0-0--1-1--1--1--1-1--1--1--1--1--1--1--1-0-1--1--1-0--1-1--1--1--1--1--1-1--1-1--1--1--1--1--1-1--1--1--1-1--1--1-1--1--1--1--—-— [1-10-10-10--10-10-0-1--1--1--1--1--1--1--1--1--1--1--1--1--1--1--1--1--1--1-1--1--1-1-1--1--1--1--1--1---1--1--1--1--1-

[ Prompt: 105.0 t/s | Generation: 46.7 t/s ]


zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --remove-orphans --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --fit off \
  --flash-attn on \
  --mlock \
  --no-warmup \
  --parallel 4 \
  --n-gpu-layers -1 \
  --n-cpu-moe 18 \
  --cache-type-k tbq4_0 \
  --cache-type-v tbq4_0 \
  --threads 12 \
  --ctx-size 128000 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --n-predict 4096 \
  -p "Why is the sky blue?"
Container turboquantamesianx-turboa-llama-cpp-run-b5c9ba13613d Creating 
Container turboquantamesianx-turboa-llama-cpp-run-b5c9ba13613d Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8702-cf2170e5f
model      : gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Why is the sky blue?

horizon-100-v-1.0-11.-10-10-0-0-0-0-0-0-0-0-0-0-0-0-0--0-0-0-0-0-0-0-0-0-0-0--10-10-0-0-0-0-0-0-0--0-0-0-0-0-0-0--

Expected result

zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --remove-orphans --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --fit off \
  --flash-attn on \
  --mlock \
  --no-warmup \
  --parallel 4 \
  --n-gpu-layers -1 \
  --n-cpu-moe 18 \
  --cache-type-k tbq3_0 \
  --cache-type-v tbq3_0 \
  --threads 12 \
  --ctx-size 128000 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --n-predict 4096 \
  -p "Why is the sky blue?"
Container turboquantamesianx-turboa-llama-cpp-run-45926ad6657a Creating 
Container turboquantamesianx-turboa-llama-cpp-run-45926ad6657a Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8702-cf2170e5f
model      : gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Why is the sky blue?

[Start thinking]
"Why is the sky blue?"
Science/Physics/Atmospheric science.
General audience (needs to be clear, accurate, but not overly dense with jargon unless explained).

    *   Sunlight (white light) is made of all colors of the rainbow (ROYGBIV).
    *   Light travels as waves.
    *   Different colors have different wavelengths (Red = long, Blue/Violet = short).
    *   Earth's atmosphere is filled with gases (Nitrogen, Oxygen).
    *   *Rayleigh Scattering:* When light hits gas molecules, shorter wavelengths (blue) scatter more easily in all directions than longer wavelengths (red).
    *   The human eye is more sensitive to blue than violet

Gemma4tbq4.txt

gemma3load.txt

[zekrom-vale]

The tbqp4_0 does the same thing

Logs

```console zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --entrypoint llama-cli turboa-llama-cpp \ --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \ --mmproj /models/mmproj-google_gemma-4-26B-A4B-it-f16.gguf \ --flash-attn on \ --mlock \ --no-warmup \ --parallel 4 \ --n-gpu-layers -1 \ --n-cpu-moe 18 \ --cache-type-k tbqp4_0 \ --cache-type-v tbqp4_0 \ --threads 12 \ --ctx-size 128000 \ --batch-size 4096 \ --ubatch-size 1024 \ --n-predict 4096 \ -p "Why is the sky blue?" Container turboquantamesianx-turboa-llama-cpp-run-89750b837243 Creating Container turboquantamesianx-turboa-llama-cpp-run-89750b837243 Created ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB): Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model...

▄▄ ▄▄
██ ██
██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
██ ██
▀▀ ▀▀

build : b8702-cf2170e5f
model : gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
modalities : text, vision

available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read add a text file
/glob add text files using globbing pattern
/image add an image file

Why is the sky blue?

---

"’ —- -- - ----- - ---- - --- - --------- - -- - --- - --- - - -- 100-1-1-1-1-1-1-1-1--1-1--1--1-1-1-1--1-1--1-1-1--1-0-1--1-1-1--1--1--1-1--1--1--1-1-1-1--1-1--1-1-1--1-1-1-1-1-—-1-1-1-1-1-1--1--1-1-1-1-1--1--1--1-1-1

[ Prompt: 88.9 t/s | Generation: 45.6 t/s ]

</details>

[zekrom-vale]

Ran some perplexity and speed tests, odd that quite a few combinations failed and not just the ones documented. And the K tbq4_0 perplexities were terrible.

1. Full Results

CTK	CTV	PPL (Lower=Better)	KV VRAM (MiB)	Prefill pp512 (t/s)	Gen tg128 (t/s)
q8_0	q8_0	7373.0368	167.34	1043.99	47.62
q8_0	tbq4_0	N/A	123.66	N/A	N/A
q8_0	tbq3_0	N/A	113.82	N/A	N/A
q8_0	tbqp3_0	N/A	N/A	N/A	N/A
tbq4_0	q8_0	15869.1594	197.49	1065.90	47.56
tbq4_0	tbq4_0	414011.4210	79.98	1062.60	46.18
tbq4_0	tbq3_0	25638.1053	70.14	997.39	46.69
tbq4_0	tbqp3_0	N/A	N/A	N/A	N/A
tbq3_0	q8_0	8515.4773	187.65	998.11	47.39
tbq3_0	tbq4_0	N/A	70.14	N/A	N/A
tbq3_0	tbq3_0	10174.0177	60.29	960.55	46.13
tbq3_0	tbqp3_0	N/A	N/A	N/A	N/A
tbqp3_0	q8_0	5228.8648	188.26	995.01	46.06
tbqp3_0	tbq4_0	N/A	70.75	N/A	N/A
tbqp3_0	tbq3_0	8374.3635	60.91	958.42	44.93
tbqp3_0	tbqp3_0	6900.2072	60.91	N/A	N/A

2. Perplexity (PPL) Matrix

CTK \ CTV	q8_0	tbq4_0	tbq3_0	tbqp3_0
q8_0	7373.0368	N/A	N/A	N/A
tbq4_0	15869.1594	414011.4210	25638.1053	N/A
tbq3_0	8515.4773	N/A	10174.0177	N/A
tbqp3_0	5228.8648	N/A	8374.3635	6900.2072

3. KV Cache VRAM Usage (MiB) Matrix

CTK \ CTV	q8_0	tbq4_0	tbq3_0	tbqp3_0
q8_0	167.34	123.66	113.82	N/A
tbq4_0	197.49	79.98	70.14	N/A
tbq3_0	187.65	70.14	60.29	N/A
tbqp3_0	188.26	70.75	60.91	60.91

4. Prefill Speed pp512 (t/s) Matrix

CTK \ CTV	q8_0	tbq4_0	tbq3_0	tbqp3_0
q8_0	1043.99	N/A	N/A	N/A
tbq4_0	1065.90	1062.60	997.39	N/A
tbq3_0	998.11	N/A	960.55	N/A
tbqp3_0	995.01	N/A	958.42	N/A

Note: For CTV tbqp3_0 is replaced with tbq3_0 automatically with some of the runs.

gemma-4-benchmark_results.txt

[AmesianX]

Thanks for the detailed report and logs!

Supported combinations

Currently TurboQuant supports the following --cache-type-k / --cache-type-v combinations:

K cache	V cache	Description
tbqp3	tbq3	Recommended. Best quality (2-bit MSE + 1-bit QJL for K, 3-bit Lloyd-Max for V)
tbqp4	tbq4	Higher bit (3-bit MSE + 1-bit QJL for K, 4-bit Lloyd-Max for V)
tbq3	tbq3	K without QJL correction
tbq4	tbq4	K without QJL correction

Important: tbqp (TurboQuant-prod) types are K-only. The p stands for the QJL (Quantized Johnson-Lindenstrauss) correction, which operates on the attention score (Q·K dot product). It has no meaning for V cache, which goes through weighted sum → IWHT decode. Setting --cache-type-v tbqp4_0 or --cache-type-v tbqp3_0 will produce garbage output.

In your case, --cache-type-k tbq4_0 --cache-type-v tbq4_0 should be a valid combination, but tbq4 has not been extensively tested on Gemma 4 yet — our primary testing has been on tbqp3/tbq3. There may be a remaining bug in the tbq4 path for head_dim=512 models.

Recommended command

llama-cli \
  --model gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --flash-attn on \
  --cache-type-k tbqp3 --cache-type-v tbq3 \
  --n-gpu-layers -1 \
  --ctx-size 128000 \
  -p "Why is the sky blue?"

Could you try tbqp3/tbq3 and report back? We haven't had the bandwidth to add input validation for invalid V-type combinations yet (been focused on kernel development), but it's on the TODO list.

For Gemma 3: it uses a different architecture and hasn't been tested with TBQ. Loading issues are expected at this point.

[zekrom-vale]

Already uploaded that with the perplexity and speed, odd that CTK tbqp3_0 CTV q8_0 beat just q8_0. While CTK tbqp3_0 CTV tbq3_0 dropes PPL by 2000 compared to CTK tbq3_0 CTV tbq3_0. Note that I cannot run it without offloading layers or experts for this size. Regardless, here is the tbqp3/tbq3 with the simple prompt:

zekromllm@zekromllm:/HomeLab/GPU/TurboQuantAmesianX$ docker compose run --rm --entrypoint llama-cli turboa-llama-cpp \
  --model /models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf \
  --flash-attn on \
  --cache-type-k tbqp3 --cache-type-v tbq3 \
  --n-gpu-layers -1 \
  --ctx-size 128000 \
  --n-gpu-layers -1 \
  --n-cpu-moe 18 \
  -p "Why is the sky blue?"
Container turboquantamesianx-turboa-llama-cpp-run-c9bdf32e2db3 Creating 
Container turboquantamesianx-turboa-llama-cpp-run-c9bdf32e2db3 Created 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15842 MiB):
  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15842 MiB

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8702-cf2170e5f
model      : gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Why is the sky blue?

[Start thinking]
"Why is the sky blue?"
Provide a clear, scientifically accurate, and easy-to-understand explanation.

    *   Sunlight (white light) is made of all colors (spectrum).
    *   Earth's atmosphere is filled with gases (nitrogen, oxygen) and particles.
    *   Rayleigh Scattering: The way light interacts with these molecules.
    *   Wavelengths: Blue/violet have short wavelengths; red/orange have long wavelengths.
    *   Scattering efficiency: Shorter wavelengths scatter more easily.
    *   Why not violet? (The sun emits more blue, and our eyes are more sensitive to blue).

    A good scientific explanation should follow a logical flow:
    *   **Step 1: Sunlight is a rainbow.** (The composition of light).
    *   **Step 2: The atmosphere is a filter/obstacle course.** (The role of the atmosphere).
    *   **Step 3: Rayleigh Scattering.** (The core mechanism).
    *   **Step 4: Why blue specifically?** (Wavelength comparison).
    *   **Step 5: The "Why not violet?" nuance.** (Addressing common follow-up questions).
    *   **Step 6: Sunsets.** (Connecting to a related phenomenon for completeness).

    *   *Step 1 (Sunlight):* Start by saying sunlight looks white but is actually a mix of all colors (ROYGBIV).
    *   *Step 2 (Atmosphere):* The atmosphere is made of molecules (mostly nitrogen and oxygen). These are tiny.
    *   *Step 3 (Scattering):* When light hits these molecules, it scatters. This is called "Rayleigh scattering."
    *   *Step 4 (Wavelengths):* Different colors have different wavelengths. Red is long/lazy; blue is short/choppy. The small gas molecules are just the right size to catch and scatter the short blue waves. The long red waves pass through mostly straight.
    *   *Step 5 (The Violet Question):* Violet has even shorter wavelengths than blue. So why isn't the sky violet? Two reasons: The Sun emits more blue than violet, and human eyes are much more sensitive to blue.
    *   *Step 6 (Sunsets):* At sunset, light has to travel through *more* atmosphere. By the time it reaches you, the blue is scattered away entirely, leaving only the reds and oranges.

    *   Use bold headings.
    *   Use bullet points for readability.
    *   Keep the tone informative but accessible.
[End thinking]

The short answer is a phenomenon called **Rayleigh scattering**. 

To understand how it works, you have to look at three things: sunlight, the Earth's atmosphere, and how light behaves as a wave.

### 1. Sunlight is a rainbow
Although sunlight looks white to us, it is actually made up of all the colors of the rainbow (red, orange, yellow, green, blue, indigo, and violet). Light travels as **waves**, and each color has a different wavelength:
*   **Red light** has long, lazy wavelengths.
*   **Blue and violet light** have short, choppy wavelengths.

### 2. The Atmosphere is an obstacle course
Earth's atmosphere is filled with gases (mostly nitrogen and oxygen). When sunlight hits the atmosphere, it strikes the molecules of these gases. 

Because these gas molecules are very small, they affect the colors differently based on their wavelengths:
*   The **long wavelengths** (reds and yellows) pass through the atmosphere mostly straight, without bumping into much. They travel from the sun to your eyes quite easily.
*   The **short wavelengths** (blues and violets) strike the gas molecules and get scattered in every direction. 

Because the blue light is bouncing around everywhere in the upper atmosphere, when you look up, your eyes are catching that scattered blue light coming from every part of the sky.

### 3. Why isn't the sky violet?
If violet light has an even shorter wavelength than blue light, you might wonder why the sky doesn't look purple. There are two main reasons:
1.  **The Sun's Output:** The Sun emits much more blue light than violet light.
2.  **Human Biology:** Our eyes are much more sensitive to blue than to violet. Our visual system interprets the mix of scattered violet and blue light as a pale blue.

---

### Bonus: Why are sunsets red?
When the sun is setting or rising, it is much lower on the horizon. This means the sunlight has to travel through a **much thicker layer of atmosphere** to reach your eyes. 

By the time the light gets to you, the blue light has been scattered away completely in other directions. The only colors left that haven't been scattered out of your line of sight are the long-wavelength colors: **reds, oranges, and pinks.**

[ Prompt: 86.6 t/s | Generation: 43.4 t/s ]

[AmesianX]

Thanks for the thorough benchmark — this is very helpful data!

Good news: tbqp3/tbq3 works correctly as shown in your "Why is the sky blue?" test. That's the recommended combination.

Regarding the high PPL numbers

The PPL values you're seeing (7373~414011) are likely caused by CPU offloading (--n-cpu-moe 18). TurboQuant runs entirely inside the CUDA flash attention kernel — it requires all KV cache operations to happen on GPU. When layers or experts are offloaded to CPU, the TBQ quantization/dequantization path is not executed correctly.

TurboQuant currently does not support CPU offloading. All layers must be on GPU (--n-gpu-layers -1 with sufficient VRAM) for correct results. On a 16GB GPU with a 26B model, CPU offloading is unavoidable, which will degrade TBQ quality significantly.

For accurate PPL comparison, the model needs to fit entirely in GPU VRAM. Our benchmarks were run on a DGX Spark (128GB VRAM) with full GPU offload, where tbqp3/tbq3 achieves PPL 445.7 (1.06x vs f16 419.8).

About tbq4_0 PPL 414011

The tbq4 path has a known bug for head_dim=512 models (Gemma 4) that was fixed after v1.5.2. Combined with CPU offloading, this explains the extreme PPL. We recommend sticking with tbqp3/tbq3 for now.

Summary

• ✅ tbqp3/tbq3 — use this, it works
• ⚠️ CPU offloading (--n-cpu-moe) — not supported, will cause degraded quality
• ❌ tbq4 on Gemma 4 — known bug, fix coming
• ❌ tbqp* as V type — not valid (QJL is K-only)

[AmesianX]

Fixed: tbq4/tbqp4 on Gemma 4 (head_dim=512)

The root cause has been identified and fixed in commit c174d3ec0.

Bug: tbq4_0 and tbqp4_0 used a 256-point WHT for K/V encoding, while the query preprocessing used a 512-point WHT for head_dim=512 models (Gemma 4). This WHT domain mismatch caused the Q·K dot product to produce meaningless values → garbage output.

Fix: Added quantize_f32_tbq4_0_block_512 and quantize_f32_tbqp4_0_block_512 functions that perform a proper 512-point WHT, matching the query side. The tbq3/tbqp3 paths already had this — it was only missing for the 4-bit variants.

Verified: tbqp4/tbq4 now produces correct output on Gemma 4 26B:

"The sky appears blue because of a phenomenon called Rayleigh scattering, where sunlight interacts with the Earth's atmosphere. As sunlight reaches the atmosphere, shorter blue wavelengths are scattered in all directions by gas molecules more effectively than other colors."

Please pull the latest main and rebuild to test. All K/V combinations are now supported:

--cache-type-k \ --cache-type-v	f16	q8_0	tbq3	tbq4
tbq3	✅	✅	✅	✅
tbq4	✅	✅	✅	✅
tbqp3	✅	✅	✅	✅
tbqp4	✅	✅	✅	✅

Reminder: tbqp types are K-only. Do not use them for --cache-type-v.

[zekrom-vale]

I figured that the PPL was not comparable to different configurations of cpu moe since it was so high. But I hope it's at least comparable to the same run.

[AmesianX]

v1.5.3 released. For Gemma 4 26B with head_dim=64, we now recommend:

--cache-type-k tbq4 --cache-type-v tbq3 --flash-attn on

This achieves 35/35 math accuracy on our benchmark. If you're running MoE layers on CPU (--n-gpu-layers < total), please ensure CUDA 12.4+ is used. Could you share your exact run command and GPU layer split?

[AmesianX]

Important clarification: TurboQuant requires:

• CUDA only (no ROCm/Metal/CPU)
• Flash attention ON (--flash-attn on)
• All layers on GPU (--n-gpu-layers 999) — MoE CPU offloading is not supported

All model layers must be fully loaded on GPU for TurboQuant to function correctly.