Add Gemma 4 attention features: QK/value norms, shared KV, partial RoPE, embedding scale, logit softcap#492

Open

jlamypoirier wants to merge 12 commits intomainfrom

Commits on Apr 30, 2026

Add embedding_scale and final_logit_softcap for Gemma 4 support

jlamypoirier
and
claude
committed
Add QK norm and post-mixer/MLP normalization for Gemma 4 support

jlamypoirier
and
claude
committed
Replace per-feature attention tests with a single combined test_attention

jlamypoirier
and
claude
committed
Add independent reference to test_attention, extend lengths

jlamypoirier
and
claude
committed
Expand test_attention with MQA/MHA/rotary/norm coverage

jlamypoirier
and
claude
committed
Add ProportionalRotary and rewrite test_rotary as parametrized suite

jlamypoirier
and
claude
committed
Add value_norm, shared_key_value, and FixedRMSNorm (Gemma 4 attention)

jlamypoirier
and
claude
committed
Fix in-place rotary corruption of query_norm backward context

jlamypoirier
and
claude
committed
Add HybridMoEMLP and independent pre-norm controls for decoder block

jlamypoirier
and
claude
committed
Add Gemma 4 HuggingFace checkpoint converter

jlamypoirier
and
claude
committed
Improve Gemma4 converter: shared_key_value, MoE weight shapes, roundtrip test

jlamypoirier
and
claude
committed
Fix test_rotary and test_mlp failures

jlamypoirier
and
claude
committed