dreambooth finetune

[Jason-kid]

hello,when using dreambooth to train anime characters, I encountered an embarrassing problem.

1. When using dreambooth lora without train_text_encoder, I get pretty bad results;
2. But when I use dreambooth with train_text_encoder, the results are pretty good.
   Well, any suggestions? lora+train_text_encoder or anything else?

[patrickvonplaten]

We sadly don't natively support LoRA + Text Encoder because we cannot hack into the text encoder well.

Maybe it's time to look for a semi-hacky solution here - @sayakpaul @williamberman what do you think?

[Jason-kid]

what about unet_lora and whole text_encoder?

[sayakpaul]

what about unet_lora and whole text_encoder?

The purpose of using LoRA is to allow consumer GPUs to run DreamBooth-like training. Of course, training the text encoder without LoRA while training the UNet with LoRA is a possibility, but it kind of defeats the purpose of "parameter-efficient fine-tuning".

Maybe it's time to look for a semi-hacky solution here - @sayakpaul @williamberman what do you think?

Agreed. I discussed some stuff with @pacman100 regarding the use of peft for fine-tuning the text encoder with LoRA. I think we can do that nicely. Let me work on a small PoC and update here.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.