AttributeError: 'Linear' object has no attribute 'set_lora_layer'

[aihao2000]

Describe the bug

I use (ttps://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py to train lora and raise an error AttributeError: Linear' object has no attribute 'set_lora_layer.

Reproduction

training examples/text_to_image/train_text_to_image_lora.py

Logs

No response

System Info

• diffusers version: 0.24.0.dev0
• Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.27
• Python version: 3.10.13
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.35.2
• Accelerate version: 0.24.1
• xFormers version: 0.0.22.post7
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @patrickvonplaten

[aihao2000]

Solved after uninstalling peft.

[johnowhitaker]

Likewise.

[patrickvonplaten]

Hmm we should not have to uninstall PEFT in order to run our example scripts. cc @sayakpaul @younesbelkada could you take a look here?

[BenjaminBossan]

I did a quick search and indeed, there are 3 examples which call set_lora_layer unconditionally:

• examples/text_to_image/train_text_to_image_lora.py (reported here)
• examples/dreambooth/train_dreambooth_lora.py
• examples/dreambooth/train_dreambooth_lora_sdxl.py

The package code itself looks good though, all instances of set_lora_layer that I could find were guarded.

[sayakpaul]

@patrickvonplaten I had a chance to investigate the problem. We have:

https://github.com/huggingface/diffusers/blob/50a749e90990932d49556ee54a333278559722f3/src/diffusers/models/attention_processor.py#L176-#L179

In case peft is present we're relying on nn.Linear, instead of LoRACompatibleLinear (which is the right thing to do). However, our training scripts rely on set_lora_layer(). I am not sure there's any other way around for this other than just using peft. #5419 should solve all of these problems quite nicely.

[Johnson-yue]

how to fixed this bug , during training lora with script must uninstall peft but during inference , dependence peft ??? It is a big problem for usage

[aihao2000]

@johnowhitaker You can try torch.load lora_weights file. Then use pipe.unet.load_attn_procs

[patrickvonplaten]

Can we add if-else statements to the example scripts just to make sure?

[sayakpaul]

Can you elaborate what you mean?

[patrickvonplaten]

Like can we make sure that we don't use PEFT in the unet if the example script doesn't use PEFT (even if peft is installed)

[sayakpaul]

Honestly, I don't think we need to do that because we know that #5388 is going to be merged very very soon. So, any changes will soon become redundant and will need to be removed.