Make Gemma4ClippableLinear inherit from nn.Linear for PEFT/LoRA compatibility

[albertorkive]

What does this PR do?

Makes Gemma4ClippableLinear inherit from nn.Linear instead of wrapping one via composition, enabling PEFT/LoRA to discover and target vision/audio encoder layers.

Problem: PEFT's LoRA module discovery uses isinstance(module, nn.Linear) to find targetable layers. The current Gemma4ClippableLinear subclasses nn.Module and stores an internal self.linear = nn.Linear(...), so PEFT skips all vision and audio encoder projections (q_proj, k_proj, v_proj, o_proj, ffw layers). Users cannot fine-tune the Gemma4 vision tower with LoRA.

Fix:

• Change Gemma4ClippableLinear to inherit from nn.Linear directly
• Weight lives as self.weight (standard nn.Linear) instead of self.linear.weight
• Clipping behavior is fully preserved
• Add _remap_legacy_keys state dict pre-hook for backward compatibility with existing checkpoints that store weight under "linear.weight"
• Update weight converter to use new key names
• Fix three .linear.weight references in forward methods

Backward compatibility: Existing checkpoints with linear.weight keys load correctly via the pre-hook remap. Verified with strict=True:

# Old checkpoint format
old_sd = {"linear.weight": ..., "input_min": ..., ...}

# Loads into new class without errors
new_module.load_state_dict(old_sd, strict=True)  # works

Files changed

• src/transformers/models/gemma4/modular_gemma4.py — source of truth
• src/transformers/models/gemma4/modeling_gemma4.py — generated, matching changes
• src/transformers/models/gemma4/convert_gemma4_weights.py — updated key names

How to reproduce the bug

from transformers import Gemma4ForConditionalGeneration
import torch.nn as nn

model = Gemma4ForConditionalGeneration.from_pretrained("google/gemma-4-12b-it")

# These are all False — PEFT can't target them
for name, mod in model.named_modules():
    if "vision_tower" in name and hasattr(mod, "linear"):
        print(f"{name}: isinstance(nn.Linear) = {isinstance(mod, nn.Linear)}")
        # Prints: False for every ClippableLinear module

After this PR, all Gemma4ClippableLinear modules pass isinstance(mod, nn.Linear).

[Rocketknight1]

cc @ArthurZucker @Cyrilvallez

[ArthurZucker]

SGTM I don't know why this was not done earlier (cc @Cyrilvallez as you worked on this might have been a reason?)

[ArthurZucker]

This should be done in the conversion_mapping with our WeightRenaming API!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: gemma4

[ArthurZucker]

better!

[ArthurZucker]

Valid if all layers used this! I did not check but we might need restriction on layer path?

[albertorkive]

No other module in the Gemma3/3n/4 tree uses self.linear as an attribute name. Should be safe.

[Cyrilvallez]

Unfortunately, we cannot do that as it fully breaks quantization! Quantization methods replace nn.Linear modules with their own, and here it would replace them but then the custom forward will added clipping would be fully lost!

[Cyrilvallez]

If you want to use peft, you need to explicitly set which modules you want to target manually. I think we had internal discussions about making that easier, cc @merveenoyan @BenjaminBossan

[BenjaminBossan]

Exactly as Cyril said, it's a matter of setting the correct target modules. Changing the parent class is not the solution.