[WIP][DRAFT] PEFT integration

[younesbelkada]

cc @sayakpaul @pacman100 @BenjaminBossan @patrickvonplaten

needs huggingface/transformers#25721

To test it, install that branch + peft main and run:

from diffusers import StableDiffusionPipeline
import os
from huggingface_hub import HfApi
import torch

path = "runwayml/stable-diffusion-v1-5"
lora_id = "takuma104/lora-test-text-encoder-lora-target"

pipe = StableDiffusionPipeline.from_pretrained(path, torch_dtype=torch.float16)
pipe.load_lora_weights(lora_id)
pipe = pipe.to("cuda")

prompt = "a red sks dog"

images = pipe(
    prompt=prompt, 
    num_inference_steps=15, 
    # cross_attention_kwargs={"scale": 0.5},
    generator=torch.manual_seed(0)
).images

for i, image in enumerate(images):
    file_name = f"aa_{i}"
    os.makedirs("images-integration", exist_ok=True)
    path = os.path.join("images-integration", f"{file_name}.png")
    image.save(path)

Currently on main branch, this should give without scale:

With scale = 0.5:

TODOs:

• Remove current LoRA logic on text encoder
• Fully remove all LoRA logic from diffusers for inference and replace it with PEFT for infenence (this will be difficult).
• Make it work with scale argument
• Again make sure that loading and saving works as well for diffusers format (not PEFT yet) and make sure that our training scripts still work as expected
• Make sure multi-alpha works as expected
• Correctly deal with merging LoRA layers
• Deprecate old diffusers serialization format and replace with peft and peft config format
• Discuss how to deal with training scripts

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

@younesbelkada @sayakpaul should we maybe first merge a PR that just removes the LoRA logic for the text encoder and only in a next PR we tackle the UNet? I feel like otherwise the PRs are getting too complex.

[younesbelkada]

Sure yes, let me open a separate PR that includes only the commits of text encoder and leave this PR for Unet

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.