Dreambooth + Lora is broken

[tcapelle]

Describe the bug

The loading of the LoRA weights is not working anymore. Reverting to previous commit fixes the issue:

https://wandb.ai/capecape/dreambooth-lora/reports/Dreambooth-LoRA-issues--Vmlldzo0MTQwODI0

I think this is related to how the LoRA weights are applied.

Reproduction

https://wandb.ai/capecape/dreambooth-lora/reports/Dreambooth-LoRA-issues--Vmlldzo0MTQwODI0

Logs

No response

System Info

• diffusers version: 0.16.0.dev0
• Platform: Linux-5.15.0-1032-gcp-x86_64-with-glibc2.31
• Python version: 3.10.8
• PyTorch version (GPU?): 1.13.1 (True)
• Huggingface_hub version: 0.13.4
• Transformers version: 4.28.1
• Accelerate version: 0.18.0
• xFormers version: 0.0.18
• Using GPU in script?: V100
• Using distributed or parallel set-up in script?: Noe

[tcapelle]

cc. @patrickvonplaten

[sayakpaul]

Currently being worked on here: #3180

Thanks for reporting and for your patience :)

[patrickvonplaten]

@tcapelle #3180 has been merged, can you try again?

[tcapelle]

Everything looks awesome, thanks @patrickvonplaten and @sayakpaul =)