[fix] PEFT integration fixes preventing save/load & integration

[tomaarsen]

What does this PR do?

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Details

I fixed 2 issues in this PR, and had an agent write some matching tests.

1. KeyError in PEFT loading for non-MoE conversion-mapped architectures

When integrating https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b into Transformers / Sentence Transformers, I ran into the following issue:

import tempfile
from peft import LoraConfig, get_peft_model
from transformers import AutoModel

BASE = "trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration"
model = AutoModel.from_pretrained(BASE)
peft_model = get_peft_model(
    model,
    LoraConfig(r=4, lora_alpha=4, target_modules=["q_proj", "v_proj"], task_type="FEATURE_EXTRACTION"),
)
with tempfile.TemporaryDirectory() as tmp:
    peft_model.save_pretrained(tmp)
    AutoModel.from_pretrained(tmp)  # KeyError: 'qwen2_vl' before this PR

Because we're loading a PEFT model, we're going through convert_peft_config_for_transformers. This again calls _convert_peft_config_moe, which seemingly uses _MODEL_TO_CONVERSION_PATTERN to determine if we need to go through with the MoE conversion:

transformers/src/transformers/integrations/peft.py

Lines 1077 to 1082 in def5e68

	def _convert_peft_config_moe(peft_config, model_type: str):
	base_model_type = _MODEL_TO_CONVERSION_PATTERN.get(model_type, None)
	if base_model_type is None:
	return peft_config
	target_module_mapping = _MOE_TARGET_MODULE_MAPPING[base_model_type]

However, _MODEL_TO_CONVERSION_PATTERN has a bunch of non-MoE architectures in it nowadays too:

transformers/src/transformers/conversion_mapping.py

Lines 63 to 77 in def5e68

	"paligemma": "llava",
	"aya_vision": "llava",
	"got_ocr2": "llava",
	"shieldgemma2": "llava",
	"gemma3": "llava",
	"internvl": "llava",
	"llava_next_video": "llava_next",
	"llava_onevision": "llava_next",
	"vipllava": "llava",
	"mistral3": "llava",
	"qwen2_5_vl": "qwen2_vl",
	"sam3_tracker_video": "sam3_tracker",
	"pp_chart2table": "llava",
	"gemma3n_text": "qwen3_5_text",
	"qwen3_5_moe_text": "qwen3_5_text",

And so _MOE_TARGET_MODULE_MAPPING[base_model_type] fails with a KeyError.

2. PEFT adapter weights dropped for base-only models

These lines here will rename the weights from base_model.model.model. to model.:

transformers/src/transformers/integrations/peft.py

Lines 227 to 238 in def5e68

	# TODO: remove once PEFT < 0.19 no longer supported
	def build_peft_weight_mapping(
	weight_conversions: list[WeightConverter | WeightRenaming] | None, adapter_name: str, peft_config=None
	) -> list[WeightConverter | WeightRenaming]:
	# We iterate over all the operations of the original model and simply edit them to apply to the PEFT adapter when
	# appropriate.
	# Note: This function is used in PEFT, changing it requires coordination.
	if not weight_conversions:
	return []
	# strip "base_model.model" and add adapter name
	new_weight_conversions = [WeightRenaming("base_model.model.model.", "model.")]

But this assumes 2 levels of model., i.e. something that only happens on models with heads, headless models (i.e. loaded with AutoModel) won't work with this anymore. I believe this is a regression introduced in v5 via #43261, and it's the causing issues on Sentence Transformers: huggingface/sentence-transformers#3701

I extended this to also replace base_model.model. with "", so that "base_model.model" gets removed (as the comment already says) even if there aren't 2 levels of model.

Who can review?

@ArthurZucker @BenjaminBossan

• Tom Aarsen

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for fixing this, Tom. Changes LGTM.

Regarding your first issue, we fixed the corresponding code in PEFT in huggingface/peft#3127 but forgot to fix it here (for context, these functions will all be imported from PEFT eventually but we currently duplicate them in transformers for backwards compatibility).

[tomaarsen]

Very glad to see that we found the same solution here. I'm looking forward to 0.19 when we can rely on peft fully for these. I'll wait for the tests to hopefully pass, and then I'll try to ping a maintainer 🤗

• Tom Aarsen

[tomaarsen]

Test failures are due to 404's on this model repository: https://huggingface.co/AI-Sweden-Models/gpt-sw3-126m
It's not indicative of any issues in transformers, and I think this should be good to go otherwise. cc @zucchini-nlp do you know if I/you can merge?

• Tom Aarsen

[zucchini-nlp]

CI was fixed yesterday, rebasing will help

I don't see anything super core, and since Benjamin agrees imo can be merged. For the second point, it is also related to base_model_prefix so tagging @yonigozlan and @vasqu (just fyi and related to internal thread, maybe Yoni's PR can fix it without regex)

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45428&sha=9b824c