Using load_textual_inversion appears to do nothing

[sakistriker]

Describe the bug

When using load_textual_inversion it does not affect inference in any way. I provided a version of the modified sample code from the docs. I am using the embedding from CivitAI as described. I run once without loading the textual inversion and once with, they produce the same image. Am I missing something? Thanks for the help.

Reproduction

from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"

# Without Textural Inversion
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")

prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
generator = torch.Generator("cuda").manual_seed(1)

image = pipe(prompt, num_inference_steps=50, generator = generator).images[0]
display(image)

# With Textural Inversion
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
pipe.load_textual_inversion("/content/charturnerv2.pt")

prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
generator = torch.Generator("cuda").manual_seed(1)

image = pipe(prompt, num_inference_steps=50, generator = generator).images[0]
display(image)

Logs

No response

System Info

• diffusers version: 0.15.0
• Platform: Linux-5.10.147+-x86_64-with-glibc2.31
• Python version: 3.9.16
• PyTorch version (GPU?): 1.13.1+cu116 (True)
• Huggingface_hub version: 0.13.4
• Transformers version: 4.28.0
• Accelerate version: 0.18.0
• xFormers version: 0.0.16
• Running on Google Colab with NVIDIA Tesla T4 GPU

[sayakpaul]

Cc: @patrickvonplaten

[SundayLX]

Same here, nothing changed when using those embeddings from civitai.

[patrickvonplaten]

Hey @sakistriker and @SundayLX,

Thanks a lot for the nice repro!
Sorry the docs were slightly incorrect here. Could you instead run the following example:

from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"

# Without Textural Inversion
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")

prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
generator = torch.Generator("cuda").manual_seed(1)

image = pipe(prompt, num_inference_steps=50, generator = generator).images[0]
display(image)

# With Textural Inversion
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
pipe.load_textual_inversion("/content/charturnerv2.pt", token="charturnerv2")

prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
generator = torch.Generator("cuda").manual_seed(1)

image = pipe(prompt, num_inference_steps=50, generator = generator).images[0]
display(image)

[patrickvonplaten]

PR to fix: #3166

[patrickvonplaten]

Note that by default the civit ai had saved the token "testTurner" instead of "charturnerv2" which then meant that the trained textual embeddings were not called with the prompt:

prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."

If however you pass:

pipe.load_textual_inversion("/content/charturnerv2.pt", token="charturnerv2")

The correct token is loaded

[patrickvonplaten]

This looks better now I believe: https://colab.research.google.com/drive/1xOtPY0bkXMrgrnwHzGW5StTyZoN7LdQM?usp=sharing

[SundayLX]

@patrickvonplaten Great, it works now. Thank you!

[sakistriker]

Thank you, that was very helpful @patrickvonplaten, and exactly what I was hoping to do.

I've come across another issue while testing EasyNegative where EasyNegative.pt loads fine but EasyNegative.safetensors causes a crash. Here's the code I was using:

from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
pipe.load_textual_inversion("/content/EasyNegative.safetensors", token="EasyNegative")

prompt = "multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
negative_prompt = "EasyNegative"
generator = torch.Generator("cuda").manual_seed(1)

image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=50, generator=generator).images[0]
display(image)

And that fails with the error: UnpicklingError: invalid load key, '"'. I'm not sure if that's a problem with the embedding itself or a potential bug.

[patrickvonplaten]

@sakistriker sorry could you open a new issue here so that it's easy for others to find (just copy-paste your comment above would be great! More than happy to answer then!)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.