support fuse/unfuse multiple lora

[AnyISalIn]

What does this PR do?

This pull request adds support for multiple LORA fuse and unfuse operations. It introduces an optional parameter lora_name in the fuse_lora/unfuse_lora functions. Here is an example:

JUPYTER EXAMPLE

prompt = "masterpiece, best quality, 1girl, at dusk"
negative_prompt = ("(low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2), "
                   "bad composition, inaccurate eyes, extra digit, fewer digits, (extra arms:1.2), large breasts")

def generate_image(pipeline, images_list):
  images_list.extend(pipeline(prompt=prompt, 
    negative_prompt=negative_prompt, 
    width=512, 
    height=768, 
    num_inference_steps=15, 
    num_images_per_prompt=1,
    generator=torch.manual_seed(0)
).images)

pipeline.load_lora_weights(".", weight_name="light_and_shadow.safetensors")
pipeline.fuse_lora(lora_scale=0.5, lora_name="light_and_shadow")

generate_image(pipeline, images_list)

pipeline.load_lora_weights(".", weight_name="more_details.safetensors")
pipeline.fuse_lora(lora_scale=0.5, lora_name="more_details")

generate_image(pipeline, images_list)

unfuse a lora

pipeline.unfuse_lora(lora_name="more_details")

generate_image(pipeline, images_list)

NOTE: If you don’t specify a lora_name in fuse_lora, it will be set to the default. If you don’t specify one in unfuse_lora, it will unfuse the last LORA.

Fixes # (https://github.com/huggingface/diffusers/issues/5032)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[patrickvonplaten]

Hey @AnyISalIn,

Sorry we're in the process of replacing all of the LoRA logic with PEFT, see:

• [core / Lora] Fix lora scale on text encoders #5087
• [PEFT / LoRA] PEFT integration - text encoder #5058
• [WIP][DRAFT] PEFT integration #4780

[AnyISalIn]

@patrickvonplaten awesome! Is Kohya Lora still supporte?

[patrickvonplaten]

@patrickvonplaten awesome! Is Kohya Lora still supporte?

Yes 100%

[leopedroso45]

Hey @AnyISalIn,

Sorry we're in the process of replacing all of the LoRA logic with PEFT, see:

• [core / Lora] Fix lora scale on text encoders #5087
• [PEFT / LoRA] PEFT integration - text encoder #5058
• [WIP][DRAFT] PEFT integration #4780

that means we have support for multiple loras? Couldnt find docs for that