Use of LoRa sliders

[Jonarod]

Is your feature request related to a problem? Please describe.
In civitai, a lot of LoRa "sliders" are popping up like this one: hair-length-slider-lora. It would allow one to play around with different weights values to apply the LoRa effect. In A1111, in the prompt, one would use different weights, using this syntax: <lora:name_of_lora:weight> so for the provided LoRa example, it would be something like this <lora:hair_length_slider_v1:5>.

I am not able to make this behaviour to work in HuggingFace.

Describe alternatives you've considered
So far, the only way I could apply new slider-weights to a LoRa is by fusing the LoRa and play with the lora_scale as if it were the slider's values.
However, this is not quite the same intent, as originally the weights are meant to be controlled from within a prompt directly, independently from the currently loaded model.

[DN6]

@Jonarod Would you be able to provide a code snippet to demonstrate how this is achieved in A1111?

[Jonarod]

Honestly I am not at all familiar with A1111's codebase as I only use its webui. As a matter of fact, this is the reason I started to look for some HuggingFace's Diffusers solution in the first place :)

[patrickvonplaten]

cc @sayakpaul @younesbelkada @pacman100 - is this something we could achieve with the new PEFT integration?

[pacman100]

Hello, I did the following:

from diffusers import StableDiffusionPipeline
import torch

model_path = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16)
_ = pipe.to("cuda")

from safetensors.torch import load_file
file = "hair_length_slider_v1.safetensors"
state_dict = load_file(file)
state_dict = {k: v.to(device="cuda", dtype=torch.float16) for k,v in state_dict.items() if torch.is_tensor(v)}

pipe.load_lora_weights(state_dict, adapter_name="hair_length_slider")

Positive scale

prompt = """photo of a redhead woman with black rim glasses in the style of <lora:hair_length_slider_v1>"""
negative_prompt = "white collar, cartoon, cgi, render, illustration, painting, drawing, bad quality, grainy, low resolution, disfigured"
lora_scale= 5
generator = torch.Generator("cuda").manual_seed(1)
image = pipe(prompt, 
             negative_prompt=negative_prompt,
             generator=generator, 
             num_inference_steps=20, 
             guidance_scale=7.5, 
             cross_attention_kwargs={"scale": lora_scale}).images[0]
image

Negative scale

prompt = """photo of a redhead woman with black rim glasses in the style of <lora:hair_length_slider_v1>"""
negative_prompt = "white collar, cartoon, cgi, render, illustration, painting, drawing, bad quality, grainy, low resolution, disfigured"
lora_scale= -5
generator = torch.Generator("cuda").manual_seed(1)
image = pipe(prompt, 
             negative_prompt=negative_prompt,
             generator=generator, 
             num_inference_steps=20, 
             guidance_scale=7.5, 
             cross_attention_kwargs={"scale": lora_scale}).images[0]
image

cross_attention_kwargs={"scale": lora_scale} controls the weightage to lora. Also, found the following helpful comment on the civitai ckpt page:

The model is trained with polarity during taring. In a single batch there are items where the model is in a positive weight, and there are items in a negative weight. It allows the model to target the difference, and only the difference, between the two concepts.

Note that I am using plain SD 1.5 as base model instead of the CyberRealistic ckpt.

[sayakpaul]

Wow, this is very commendable!

So, what I am seeing here is that diffusers can natively support this if I am not wrong. Also cc @apolinario here because we were talking about it a couple days back.

[apolinario]

Hey @Jonarod , under the hood AUTOMATIC1111 is taking the LoRA information on your prompt (<lora:name_of_lora:weight>) and multiplying it to the model with a weight, just like it is possible in diffusers with a lora_scale

As you have alluded, you can apply a LoRA scale when fusing a LoRA into the Unet. While this makes inference fast, is not super practical for this use-case as you'd need to unfuse and fuse again with another scale.

To swap the lora_scale in inference time, following what @pacman100 has described should land at similar results on what you get by including the LoRA on your prompts with AUTO1111, as that is more of a UI convenience thing rather than a feature that is really included in the prompts

[Jonarod]

I see.
Very thankful for the prompt support, really appreciated.

I don't know the actual HF philosophy on this one, is it considered the final solution for this use case?
Or should I leave this feature opened until the Lora scaling can be done from the prompt directly?

A solution I could imagine would be to, for example, parse the prompt with some regex first to actually extract the Lora scale, and then automatically apply the relevant kwargs as described above.

So I'll leave it opened on my end, but feel free to close it if that does not correspond to HuggingFace's roadmap :)

[apolinario]

I think it's not super likely that adding LoRAs directly as a prompt and parsing that will be a core feature in the diffusers library pipelines.

However, you or anyone in the community is encouraged to contribute a community pipeline that does that: https://github.com/huggingface/diffusers/tree/main/examples/community

[Jonarod]

I was able to reproduce the scaling without fuse using cross_attention_kwargs={"scale": lora_scale} as demonstrated. Again, thanks for pointing that out.

However the disadvantage of this method is that scaling applies for all LoRas loaded. It is not possible to have multiple scales for multiple LoRas like in classic webUI solutions (be it A1111, ComfyUI, ...).
Also, the SOTA seems to adopt the <lora:name:scale> pattern as is visible in major models zoos like Civitai and others. Since some time now, a lot of new images have emerged with prompts similar to this:

photo of a redhead woman with black rim glasses <lora:add_detail:1> <lora:weight_slider_v2:-1> <lora:hair_length_slider_v1:2>

which is becoming the standard right now.

Finally, fusing/merging LoRas with checkpoints and/or upcoming serialisation of torch.compile()'d models, would make the final user lose the ability to further modulate the scale of the "LoRa_sliders" once loaded (since so far we only have 1 global scale argument for all LoRas). Hence IMHO, offloading LoRa scaling into prompts does not seem so much of a bad idea, and maybe that's why it's getting so popular too.

[sayakpaul]

Hi @Jonarod. Thanks for your interest.

We like to keep prompting solely to prompting. This is why, for prompt weighting, we refer users to compel. See: https://huggingface.co/docs/diffusers/main/en/using-diffusers/weighted_prompts.

So, this kind of offloading is a bit of an anti-pattern for us.

However, with the peft integration (that is upcoming), using multiple LoRAs with multiple scales is going to be much easier. See here and here.

I hope that makes sense.

[Jonarod]

So if I read correctly, the new PR will address the need for multiple LoRa adapters with this solution:

pipe.set_adapters(["pixel", "toy"], adapter_weights=[0.5, 1.0])

Then if I am correct, it is up to the developer to load/unload adapters between inferences.

Cool with me, looking forward to this new PR !!
Thank you all for your support !! 🙏🏼

[sayakpaul]

Then if I am correct, it is up to the developer to load/unload adapters between inferences.

Exactly! And again, thanks for the in-depth discussions! As maintainers, we LOVE them.