Error when load LoRA weight after training dreambooth_lora

[CaffreyR]

Describe the bug

Dear All,
I have use the script and md file to train dreambooth_LoRA.py and I try to use the method you inference. https://github.com/huggingface/diffusers/tree/main/examples/dreambooth

Reproduction

This is the repo I use after the update of train_dreambooth_lora.py

Probably because of the PEFT

from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
import torch

pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")

pipe.unet.load_attn_procs("My Repo")

Logs

Traceback (most recent call last):
  File "test.py", line 32, in <module>
    pipe.unet.load_attn_procs("My repo")
  File "/anaconda3/envs/bias/lib/python3.9/site-packages/diffusers/loaders/unet.py", line 264, in load_attn_procs
    rank = value_dict["lora.down.weight"].shape[0]
KeyError: 'lora.down.weight'

System Info

• diffusers version: 0.24.0
• Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
• Python version: 3.9.18
• PyTorch version (GPU?): 1.11.0+cu115 (True)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.36.1
• Accelerate version: 0.25.0
• xFormers version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@sayakpaul @patrickvonplaten @DN6 @yiyixuxu @sayakpaul

[CaffreyR]

Same issues #6130

[CaffreyR]

And it seems that pipe.unet.load_attn_procs() cannot load the weight well and the train_dreambooth_lora.py and @sayakpaul explained in #6130 is using load_lora_weights it works differently when apply to the same seed. And the load_lora_weights cannot reproduce the model card as well.

[sayakpaul]

That is an issue we're working on to fix. Refer here: #6135

[CaffreyR]

Dear @sayakpaul , are we able to use your request? It seems not merge into the example/dreambooth main. Thanks

[sayakpaul]

Yes, feel free to use the branch from #6135. It will soon be merged into main.

[CaffreyR]

Hi @sayakpaul , but it seems not working well when you use pipe.unet.load_attn_procs() while load_lora_weights works well.

[sayakpaul]

Sorry for that but load_attn_procs() will soon be deprecated and I encourage you to try out load_lora_weights() unless you have a specific reason not to. Then we're also working on #5958.

[CaffreyR]

Thank you @sayakpaul , I can use load_lora_weights() now even if #5958 is not prepared well right?

[sayakpaul]

Yeah.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.