failed to use the feature of supporting for A1111 LoRA

[icech]

I am glad to see the diffusers add the supporting for A1111 LoRA. However, i failed to ues this feature after I update the diffusers.
It did not exporting error as before, but the lora have no effect in the generated images.
I use it as following:

pipe = StableDiffusionPipeline.from_pretrained(repo_id, torch_dtype=torch.float16, 
                                               revision="fp16", safety_checker=None).to("cuda")
pipe.load_lora_weights("./Lora", weight_name="xxx.safetensors")
generator = torch.Generator(device='cuda')
prompt = "a photograph of a man running with dog"
image = pipe(prompt, width=768, height=1280, 
                       generator=generator, num_inference_steps=20, 
                       cross_attention_kwargs={"scale": 1}
                       ).images[0]

and this is how I used to add lora in the past which is form #3064 by @pdoane :

def load_lora_weights(pipeline, checkpoint_path, multiplier, device, dtype):
    LORA_PREFIX_UNET = "lora_unet"
    LORA_PREFIX_TEXT_ENCODER = "lora_te"
    # load LoRA weight from .safetensors
    state_dict = load_file(checkpoint_path, device=device)

    updates = defaultdict(dict)
    for key, value in state_dict.items():
        # it is suggested to print out the key, it usually will be something like below
        # "lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight"

        layer, elem = key.split('.', 1)
        updates[layer][elem] = value

    # directly update weight in diffusers model
    for layer, elems in updates.items():

        if "text" in layer:
            layer_infos = layer.split(LORA_PREFIX_TEXT_ENCODER + "_")[-1].split("_")
            curr_layer = pipeline.text_encoder
        else:
            layer_infos = layer.split(LORA_PREFIX_UNET + "_")[-1].split("_")
            curr_layer = pipeline.unet

        # find the target layer
        temp_name = layer_infos.pop(0)
        while len(layer_infos) > -1:
            try:
                curr_layer = curr_layer.__getattr__(temp_name)
                if len(layer_infos) > 0:
                    temp_name = layer_infos.pop(0)
                elif len(layer_infos) == 0:
                    break
            except Exception:
                if len(temp_name) > 0:
                    temp_name += "_" + layer_infos.pop(0)
                else:
                    temp_name = layer_infos.pop(0)

        # get elements for this layer
        weight_up = elems['lora_up.weight'].to(dtype)
        weight_down = elems['lora_down.weight'].to(dtype)
        alpha = elems['alpha']
        if alpha:
            alpha = alpha.item() / weight_up.shape[1]
        else:
            alpha = 1.0

        # update weight
        if len(weight_up.shape) == 4:
            curr_layer.weight.data += multiplier * alpha * torch.mm(weight_up.squeeze(3).squeeze(2), weight_down.squeeze(3).squeeze(2)).unsqueeze(2).unsqueeze(3)
        else:
            curr_layer.weight.data += multiplier * alpha * torch.mm(weight_up, weight_down)

    return pipeline

Is it my incorrect usage or is there a difference between the current code and what @pdoane provided?

[patrickvonplaten]

cc @sayakpaul

[sayakpaul]

Hi @icech.

Could you share your LoRA file so that we can debug it on our end?

Cc: @takuma104

[icech]

Hi @icech.

Could you share your LoRA file so that we can debug it on our end?

Cc: @takuma104

The link of my LoRA file is
https://civitai.com/models/7501/vivid-watercolors-lora-extraction

[sayakpaul]

@icech I am able to use the LoRA without any problems. See my Colab:
https://colab.research.google.com/gist/sayakpaul/1fff0ff9c5a059364f80ac0b64920592/scratchpad.ipynb

Of course, I don't know about the base pipeline associated with that. So, you will have to work that one out.

[icech]

@icech I am able to use the LoRA without any problems. See my Colab: https://colab.research.google.com/gist/sayakpaul/1fff0ff9c5a059364f80ac0b64920592/scratchpad.ipynb

Of course, I don't know about the base pipeline associated with that. So, you will have to work that one out.

I'm sorry for the delayed response. I have reviewed your Colab notebook and made some modifications. I can provide a comparison between the version with and without Lora, and generate four images. Only one of the images will be different.
The code is below

import torch

from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

pipeline = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, safety_checker=None
).to("cuda")

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
    pipeline.scheduler.config, use_karras_sigmas=True
)


images = pipeline(prompt="masterpiece, best quality, mountain landscape",
    negative_prompt="bad quality",
    width=512,
    height=512,
    num_inference_steps=15,
    num_images_per_prompt=4,
    generator=torch.manual_seed(0)
).images


for i, image in enumerate(images):
    image.save(f'images/nonelora_{i}.png')
    
    
pipeline.load_lora_weights(".", weight_name="vividWatercolors_10.safetensors")
images = pipeline(prompt="masterpiece, best quality, mountain landscape",
    negative_prompt="bad quality",
    width=512,
    height=512,
    num_images_per_prompt=4,
    num_inference_steps=15,
    generator=torch.manual_seed(0)
).images

for i, image in enumerate(images):
    image.save(f'images/lora_{i}.png')

[sayakpaul]

I see what you're saying. Reproduced here: https://colab.research.google.com/gist/sayakpaul/b645715d9144a3a6dc40c93bdceee929/scratchpad.ipynb.

Some questions:

• Could you also have some images (LoRA) for us so that we can have some one-on-one comparisons?
• Are the results being affected because we're not likely using the right base model? For example, here, we're using the right base model to load the parameters into. We need to ensure the base model being used here is indeed the correct one.

[takuma104]

That's quite an interesting result. It seems that the effect varies in magnitude, but it's not just the first one that differs; there appears to be some change in all of them. I've posted an image created by merging the two results using the difference mode in Photoshop.

Without LoRA:

With LoRA:

Diff:

[icech]

I see what you're saying. Reproduced here: https://colab.research.google.com/gist/sayakpaul/b645715d9144a3a6dc40c93bdceee929/scratchpad.ipynb.

Some questions:

• Could you also have some images (LoRA) for us so that we can have some one-on-one comparisons?
• Are the results being affected because we're not likely using the right base model? For example, here, we're using the right base model to load the parameters into. We need to ensure the base model being used here is indeed the correct one.

I've seen the graph in your Colab. they are the same as what I drew before. I'm sorry that I can't provide any pictures these days since I'm on vacation and don't have access to my machine. However, you can test the 'load_lora_weights(pipeline, checkpoint_path, multiplier, device, dtype)' I originally mentioned to replace 'pipeline.load_lora_weights' for loading lora. The usage is 'pipe = load_lora_weights(pipe, lora_path, 1.0, 'cuda', torch.float32)', so you can make a comparison. Based on my previous experience, the two should not be consistent, and the version of @pdoane is the expected result, consistent with A111.

[icech]

That's quite an interesting result. It seems that the effect varies in magnitude, but it's not just the first one that differs; there appears to be some change in all of them. I've posted an image created by merging the two results using the difference mode in Photoshop.

Without LoRA:

With LoRA:

Diff:

Your analysis is very rigorous. Indeed, there are differences in the subsequent images, but these differences are not as expected.

[sayakpaul]

I will dive more to find out what we're missing :) But expect some delay as I am on the move and away for sometime.

[alexblattner]

@icech use this:
loraLoader.txt

thank me later.

[icech]

@icech use this: loraLoader.txt

thank me later.

I've tried this and I know it's feasible, but I mainly want to use the official API for easier maintenance in the future. Thank you anyway.

[patrickvonplaten]

Let's try to fix this this week so it's in the next release cc @sayakpaul , could this maybe be fixed with: #3778 ?

[sayakpaul]

I originally mentioned to replace 'pipeline.load_lora_weights' for loading lora. The usage is 'pipe = load_lora_weights(pipe, lora_path, 1.0, 'cuda', torch.float32)', so you can make a comparison. Based on my previous experience, the two should not be consistent, and the version of @pdoane is the expected result, consistent with A111.

@icech I am trying to understand this better and would appreciate your inputs here. If I do pipeline.load_lora_weights(".", weight_name="vividWatercolors_10.safetensors") (as done in my Colab), it actually uses "cuda" along with a LoRA scale of 1. Is it not what you used in your experiments too?

Regardless, I will dive deeper into the loaded parameters and see what we're missing out on :)

[icech]

I originally mentioned to replace 'pipeline.load_lora_weights' for loading lora. The usage is 'pipe = load_lora_weights(pipe, lora_path, 1.0, 'cuda', torch.float32)', so you can make a comparison. Based on my previous experience, the two should not be consistent, and the version of @pdoane is the expected result, consistent with A111.

@icech I am trying to understand this better and would appreciate your inputs here. If I do pipeline.load_lora_weights(".", weight_name="vividWatercolors_10.safetensors") (as done in my Colab), it actually uses "cuda" along with a LoRA scale of 1. Is it not what you used in your experiments too?

Regardless, I will dive deeper into the loaded parameters and see what we're missing out on :)

This is same as my experiments. I will provide some images of my result tomorrow(about 10 hours later) for you to compare.