🚨 [v5] Refactor RoPE for layer types

[zucchini-nlp]

What does this PR do?

This PR enables rope layers to compute different frequencies for different layer types, which will help us to support models like ModernBert without monkey patching config on-the-fly

Main changes:

• In config classes the rope_parameters is a required attribute if model has RoPE layers. The attr must be a dict containing rope_theta and optionally other parameters to configure rope. In case we want different params per layer type, it should be a nested dict of format {"full_attn": {**rope_params}, "sliding_attn": {**different_rope_params}}
• The config attr rope_scaling is deprecated in favor of rope_parameters and raises warning. The latter name is more descriptive
• Default rope freq computation is moved to the model definition similar to eager_attention_forward, and copied with modular in each file
• RoPE layer now looks for layer types in the config and computes inv_freq for each type. If the given layer types has no rope parameters saved in config (e.g. config.rope_scaling has no key=="sliding_window") we raise an error
• All models copy from rope layers llama when possible, so that changing one file will update it everywhere. Models with layer types copy from gemma2
• Config classes now have typing hint in all language models and the rope scaling attribute is typed with TypedDict. It will make our lives easier when we decide to enforce strict type validation on configs

The changes are BC and we will support old-format config files, and standardize it when initializing the config class. The best way to review is to start from modeling_rope_utils.py -> all llama model files -> gemma2 and gemma33 model files -> tests

[ArthurZucker]

🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: apertus, arcee, aria, bamba, bitnet, blt, chameleon

[zucchini-nlp]

Just to be sure, let the slow CI run on some important models

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/gemma', 'models/gemma3', 'models/gpt2', 'models/lfm2_vl', 'models/llama', 'models/llava', 'models/mistral', 'models/olmo', 'models/paligemma', 'models/qwen2_5_omni', 'models/qwen2_vl']
quantizations: [] ...

[zucchini-nlp]

🤞🏻

[BakerBunker]

This PR caused QwenLM/Qwen3-Omni#93

[ydshieh]

cc @zucchini-nlp