Fix PEFT x MoEs

[ArthurZucker]

What does this PR do?

Fixes #42491

This should serve as an example of how the weight loader can be re-used in other project.
The content is probably gonna be upstreamed to peft!
Current status:

What to expect:

1. You already loaded a transformers model
2. You want to load only the Peft adapters:
   a. we check the weight conversion mapping
   b. if there are ops, we replace them with mapped peft ops
   c. we just collect lora_A and lora_B together, process them like so:

class PeftConcatenate(Concatenate):
    """Convert per-expert LoRA weights to merged MoE weights using SVD."""
    @torch.no_grad
    def convert(
        self, input_dict: dict[str, list[torch.Tensor]], source_patterns: list[str], target_patterns: list[str], **kwargs
    ) -> dict[str, list[torch.Tensor]]:
        lora_a_out = []
        lora_b_out = []
        for k,v in input_dict.items():
            if "lora_A" in k:
                lora_a_out.append(v)
            elif "lora_B" in k:
                lora_b_out.append(v)
        lora_a_out = torch.cat(lora_a_out, dim=0)
        for i in range(len(lora_b_out)):
            lora_b_out.append(torch.block_diag(lora_b_out[0][i], lora_b_out[1][i]))
        lora_b_out = torch.stack(lora_b_out[2:], dim=0)
        return {
            target_patterns[0]+".lora_A.weight": [lora_a_out],
            target_patterns[0]+".lora_B.weight": [lora_b_out],
        }

    d. The output fused gate_up.lora_A/B are loaded in the model

CF: (Credits to @BenjaminBossan for the pic)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Just leaving some general comments on this PR:

1. I wouldn't override __setattr__ in WeightTransform, as it's error prone. Instead, change source_patterns and target_patterns to @property with an appropriate @setter and move the logic there.
2. The hotswap code path has been removed from load_adapter, it needs to be added back in.
3. Similarly, there was a if peft_config.inference_mode: self.eval() call there that's now missing.
4. As discussed, the currently hard-coded Mixtral conversion ops need to be moved to a mapping that is only called for Mixtral.
5. We should add some sanity checks, e.g. if the expert layer is targeted and the PEFT adapter is not LoRA, raise a helpful error message.

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43261&sha=92d0fa

[ArthurZucker]

Ready!