StableDiffusionControlNetInpaintPipeline `strength=1` doesn't work

[undernf]

Describe the bug

cc @williamberman

Lets say you have an image with a subject and a background, you put a mask over the background to repaint it and set the strength=1, so in theory the background should be completely ignored. But this isn't what happens, the background carries some weight in the regeneration.

It's only subtle but it's more obvious if you use, for example, an image with a pink background and then try to regenerate a new background which is usually not pink (e.g. a forest).

I don't know enough about how this works but I can see from the below example that some information from the background is being passed to the inpainted area when it shouldn't.

Reproduction

The below code uses img_red_bg_url and the output will have a red BG (which shouldn't happen because strength=1), if you change it to use img which has a pink BG, the output will also have a pink BG, again proving the original BG has some weighting in the generation.

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)

pipe = DiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    custom_pipeline="stable_diffusion_controlnet_inpaint_img2img",
    controlnet=controlnet,
    safety_checker=None,
    torch_dtype=torch.float16
).to(device)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

img_url = "https://i.ibb.co/0ZK7yL0/img.png"
img_red_bg_url = "https://i.ibb.co/zHHZTkZ/img-red-bg.png"
canny_url = "https://i.ibb.co/rp9FYCX/canny.png"
mask_url = "https://i.ibb.co/FK4DNNK/mask.png"

def decode_image(image_url):
    print("Decode image: " + image_url)
    req = urllib.request.urlopen(image_url)
    arr = np.array(bytearray(req.read()), dtype=np.uint8)
    imageBGR = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
    imageRGB = cv2.cvtColor(imageBGR, cv2.COLOR_BGRA2RGB)

    return Image.fromarray(imageRGB)

img = decode_image(img_url)
img_red_bg = decode_image(img_red_bg_url)
canny = decode_image(canny_url)
mask = decode_image(mask_url)

with autocast("cuda"), torch.inference_mode():
    output = pipe(
        prompt="product in a forest",
        negative_prompt="",
        image=img_red_bg,
        mask_image=mask,
        controlnet_conditioning_image=canny,
        height=512,
        width=512,
        num_images_per_prompt=1,
        num_inference_steps=20,
        guidance_scale=7.5,
        strength=1
    )

    for image in output.images:
        display(image)

Logs

No response

System Info

• diffusers version: 0.17.0.dev0
• Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.31
• Python version: 3.10.10
• PyTorch version (GPU?): 1.13.1+cu117 (True)
• Huggingface_hub version: 0.14.1
• Transformers version: 4.27.3
• Accelerate version: 0.17.1
• xFormers version: 0.0.16
• Using GPU in script?: y
• Using distributed or parallel set-up in script?: n

[patrickvonplaten]

This is not related to: #3532 no?

[patrickvonplaten]

Hey @undernf,

Thanks for the issue! I think the problem was simply that a community pipeline was used here which is not tested very well. If you use the official inpainting pipeline it should work just fine.

You can try this code:

#!/usr/bin/env python3
from diffusers import ControlNetModel, DDIMScheduler, DiffusionPipeline, StableDiffusionControlNetInpaintPipeline
import torch
import cv2
import numpy as np
import urllib
from PIL import Image

device = "cuda"

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)

pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    controlnet=controlnet,
    safety_checker=None,
    torch_dtype=torch.float16
).to(device)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

img_url = "https://i.ibb.co/0ZK7yL0/img.png"
img_red_bg_url = "https://i.ibb.co/zHHZTkZ/img-red-bg.png"
canny_url = "https://i.ibb.co/rp9FYCX/canny.png"
mask_url = "https://i.ibb.co/FK4DNNK/mask.png"

def decode_image(image_url):
    print("Decode image: " + image_url)
    req = urllib.request.urlopen(image_url)
    arr = np.array(bytearray(req.read()), dtype=np.uint8)
    imageBGR = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
    imageRGB = cv2.cvtColor(imageBGR, cv2.COLOR_BGRA2RGB)

    return Image.fromarray(imageRGB)

img = decode_image(img_url)
img_red_bg = decode_image(img_red_bg_url)
canny = decode_image(canny_url)
mask = decode_image(mask_url)

with torch.inference_mode():
    output = pipe(
        prompt="product in a forest",
        negative_prompt="",
        image=img_red_bg,
        mask_image=mask,
        control_image=canny,
        height=512,
        width=512,
        num_images_per_prompt=1,
        num_inference_steps=20,
        guidance_scale=7.5,
        strength=1
    )

    for image in output.images:
        image.show()

Let me know if this solves your issue!

[kadirnar]

Hey @undernf,

Thanks for the issue! I think the problem was simply that a community pipeline was used here which is not tested very well. If you use the official inpainting pipeline it should work just fine.

You can try this code:

#!/usr/bin/env python3
from diffusers import ControlNetModel, DDIMScheduler, DiffusionPipeline, StableDiffusionControlNetInpaintPipeline
import torch
import cv2
import numpy as np
import urllib
from PIL import Image

device = "cuda"

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)

pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    controlnet=controlnet,
    safety_checker=None,
    torch_dtype=torch.float16
).to(device)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

img_url = "https://i.ibb.co/0ZK7yL0/img.png"
img_red_bg_url = "https://i.ibb.co/zHHZTkZ/img-red-bg.png"
canny_url = "https://i.ibb.co/rp9FYCX/canny.png"
mask_url = "https://i.ibb.co/FK4DNNK/mask.png"

def decode_image(image_url):
    print("Decode image: " + image_url)
    req = urllib.request.urlopen(image_url)
    arr = np.array(bytearray(req.read()), dtype=np.uint8)
    imageBGR = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
    imageRGB = cv2.cvtColor(imageBGR, cv2.COLOR_BGRA2RGB)

    return Image.fromarray(imageRGB)

img = decode_image(img_url)
img_red_bg = decode_image(img_red_bg_url)
canny = decode_image(canny_url)
mask = decode_image(mask_url)

with torch.inference_mode():
    output = pipe(
        prompt="product in a forest",
        negative_prompt="",
        image=img_red_bg,
        mask_image=mask,
        control_image=canny,
        height=512,
        width=512,
        num_images_per_prompt=1,
        num_inference_steps=20,
        guidance_scale=7.5,
        strength=1
    )

    for image in output.images:
        image.show()

Let me know if this solves your issue!

This is great. How can I use controlnet depth and inpaint model together?

[sayakpaul]

@kadirnar I think this question is slightly unrelated to the PR and therefore, I suggest using our forum to ask it:
https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63

[sayakpaul]

@undernf if #3582 (comment) resolves your question, could we close the issue?

[undernf]

@patrickvonplaten and do you know how I could manually generate latents for that pipe?

I've tried this:

latents = torch.randn(
    (1, pipe.unet.in_channels, 512 // 8, 512 // 8),
    generator=generator,
    device=device,
)

But I get this error: RuntimeError: Given groups=1, weight of size [320, 4, 3, 3], expected input[8, 9, 64, 64] to have 4 channels, but got 9 channels instead

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.