[ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL #4694

kfzyqin · 2023-08-21T11:00:50Z

Overview:

This PR introduces the implementation of the inference pipeline for ControlNet with SDXL and inpainting.

Files Modified/Added:

Inference Pipeline: srcs/pipelines/controlnet/pipeline_control_inpaint_sd_xl.py
- This file contains the main implementation of the inference pipeline for ControlNet with SDXL and inpainting.
Unit Test: tests/pipelines/controlnet/test_controlnet_inpaint_sdx.py
- This file provides the unit tests to ensure the correct functionality and robustness of the implemented pipeline.

Visualizations:

To better understand the impact and functionality of the implemented pipeline, the following visualizations are provided:

Input Image
Mask
Output Image

Overview:

This PR introduces the implementation of the inference pipeline for ControlNet with SDXL and inpainting.

Files Modified/Added:

Inference Pipeline: srcs/pipelines/controlnet/pipeline_control_inpaint_sd_xl.py
- This file contains the main implementation of the inference pipeline for ControlNet with SDXL and inpainting.
Unit Test: tests/pipelines/controlnet/test_controlnet_inpaint_sdx.py
- This file provides the unit tests to ensure the correct functionality and robustness of the implemented pipeline.

Example Usage

import torch 
from PIL import Image
from transformers import DPTForDepthEstimation, DPTFeatureExtractor
import numpy as np 
import cv2 


def get_depth_map(image):
    depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")
    feature_extractor = DPTFeatureExtractor.from_pretrained("Intel/dpt-hybrid-midas")
    image = feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")
    with torch.no_grad(), torch.autocast("cuda"):
        depth_map = depth_estimator(image).predicted_depth

    depth_map = torch.nn.functional.interpolate(
        depth_map.unsqueeze(1),
        size=(512, 512),
        mode="bicubic",
        align_corners=False,
    )
    depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_map = (depth_map - depth_min) / (depth_max - depth_min)
    image = torch.cat([depth_map] * 3, dim=1)

    image = image.permute(0, 2, 3, 1).cpu().numpy()[0]
    image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))
    return image

def inpaint_with_controlnet():
    import torch
    from diffusers import StableDiffusionXLInpaintPipeline
    from diffusers.utils import load_image
    from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
    from diffusers import StableDiffusionXLControlNetInpaintPipeline

    img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
    mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

    controlnet = [
        # ControlNetModel.from_pretrained(
        #     "diffusers/controlnet-depth-sdxl-1.0", use_auth_token=True, torch_dtype=torch.float32
        # ), 
        ControlNetModel.from_pretrained(
            "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float32
        ),
    ]

    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0", 
        controlnet=controlnet,
        torch_dtype=torch.float32, 
    )
    pipe.to("cuda")

    init_image = load_image(img_url).convert("RGB")
    depth_image = get_depth_map(init_image)
    
    canny_image = np.array(init_image)

    low_threshold = 100
    high_threshold = 200

    canny_image = cv2.Canny(canny_image, low_threshold, high_threshold)

    # zero out middle columns of image where pose will be overlayed
    zero_start = canny_image.shape[1] // 4
    zero_end = zero_start + canny_image.shape[1] // 2
    canny_image[:, zero_start:zero_end] = 0

    canny_image = canny_image[:, :, None]
    canny_image = np.concatenate([canny_image, canny_image, canny_image], axis=2)
    canny_image = Image.fromarray(canny_image).resize((1024, 1024))
    
    mask_image = load_image(mask_url).convert("RGB")
    
    original_width, original_height = init_image.size
    new_width = int(original_width / 2)
    new_height = int(original_height / 2)
    init_image = init_image.resize((new_width, new_height))
    mask_image = mask_image.resize((new_width, new_height))
    depth_image = depth_image.resize((new_width, new_height))
    canny_image = canny_image.resize((new_width, new_height))
    
    prompt = "black cat with green eyes"
    strength=1.0
    controlnet_conditioning_scale = 0.3

    depth_image.save('control_image.jpg')
    image = pipe(
        prompt=prompt,
        image=init_image,
        mask_image=mask_image,
        control_image=[depth_image],
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        strength=strength,
        width=1024, 
        height=1024, 
    ).images[0]

    image.save('result_sdxl_inpaint.jpg')
    
    
if __name__ == "__main__":
    inpaint_with_controlnet()

Features

Support MultiControlNet
Compatible with new HF code

Cathy0908 · 2023-08-22T10:00:31Z

Wow, I really need it. Can it work now? I always generate black pictures with it ? Can you post the api usage, thanks a lot !

kfzyqin · 2023-08-22T10:15:52Z

Wow, I really need it. Can it work now? I always generate black pictures with it ? Can you post the api usage, thanks a lot !

I discovered some issues today, but it should generate sensible images, rather than black ones ...

Let me complete this by this week.

Feel free to add my discord: harutatsuakiyama

…sers into sdxl_ctrl_inpaint

kfzyqin · 2023-08-22T22:20:48Z

Wow, I really need it. Can it work now? I always generate black pictures with it ? Can you post the api usage, thanks a lot !

I fixed the issue yesterday. The code should work as expected.

Cathy0908 · 2023-08-23T03:47:55Z

I use the following pipeline, but still generate black image.
And I replace StableDiffusionXLControlNetInpaintPipeline with StableDiffusionXLInpaintPipeline, it works well.
Is there something wrong with my code?

def inpaint_with_controlnet():
    import torch
    from diffusers import StableDiffusionXLInpaintPipeline
    from diffusers.utils import load_image
    from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
    from pipeline_controlnet_inpaint_sd_xl import StableDiffusionXLControlNetInpaintPipeline

    img_url = "https://user-images.githubusercontent.com/8084808/262496067-e01fb3c9-aece-4560-ae64-6354fdd789d7.png"
    mask_url = "https://user-images.githubusercontent.com/8084808/262496139-234e0049-43ab-415b-ae6d-4cbb96055f6d.png"
    control_image_url = img_url

    # Compute openpose conditioning image.
    from controlnet_aux import OpenposeDetector
    openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
    control_image = openpose(load_image(control_image_url))

    controlnet = ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float16)

    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0", 
        controlnet=controlnet,
        torch_dtype=torch.float16, 
    )
    pipe.to("cuda")

    init_image = load_image(img_url).convert("RGB")
    mask_image = load_image(mask_url).convert("RGB")

    prompt = "hand"
    strength=0.5
    controlnet_conditioning_scale = 1.0

    image = pipe(
        prompt=prompt,
        image=init_image,
        mask_image=mask_image,
        control_image=control_image,
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        strength=strength,
    ).images[0]

    image.save('result.jpg')

kfzyqin · 2023-08-23T11:13:43Z

def inpaint_with_controlnet():
    import torch
    from diffusers import StableDiffusionXLInpaintPipeline
    from diffusers.utils import load_image
    from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
    from pipeline_controlnet_inpaint_sd_xl import StableDiffusionXLControlNetInpaintPipeline

    img_url = "https://user-images.githubusercontent.com/8084808/262496067-e01fb3c9-aece-4560-ae64-6354fdd789d7.png"
    mask_url = "https://user-images.githubusercontent.com/8084808/262496139-234e0049-43ab-415b-ae6d-4cbb96055f6d.png"
    control_image_url = img_url

    # Compute openpose conditioning image.
    from controlnet_aux import OpenposeDetector
    openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
    control_image = openpose(load_image(control_image_url))

    controlnet = ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float16)

    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0", 
        controlnet=controlnet,
        torch_dtype=torch.float16, 
    )
    pipe.to("cuda")

    init_image = load_image(img_url).convert("RGB")
    mask_image = load_image(mask_url).convert("RGB")

    prompt = "hand"
    strength=0.5
    controlnet_conditioning_scale = 1.0

    image = pipe(
        prompt=prompt,
        image=init_image,
        mask_image=mask_image,
        control_image=control_image,
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        strength=strength,
    ).images[0]

    image.save('result.jpg')

Thank you for the code! You need to use torch.float32 instead of torch.float16. I tested the following code, should work:

def inpaint_with_controlnet():
    import torch
    from diffusers import StableDiffusionXLInpaintPipeline
    from diffusers.utils import load_image
    from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
    from diffusers import StableDiffusionXLControlNetInpaintPipeline

    img_url = "https://user-images.githubusercontent.com/8084808/262496067-e01fb3c9-aece-4560-ae64-6354fdd789d7.png"
    mask_url = "https://user-images.githubusercontent.com/8084808/262496139-234e0049-43ab-415b-ae6d-4cbb96055f6d.png"
    control_image_url = img_url

    # Compute openpose conditioning image.
    from controlnet_aux import OpenposeDetector
    openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
    control_image = openpose(load_image(control_image_url))

    controlnet = ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float32)

    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0", 
        controlnet=controlnet,
        torch_dtype=torch.float32, 
    )
    pipe.to("cuda")

    init_image = load_image(img_url).convert("RGB")
    mask_image = load_image(mask_url).convert("RGB")
    
    original_width, original_height = init_image.size
    new_width = int(original_width / 2)
    new_height = int(original_height / 2)
    init_image = init_image.resize((new_width, new_height))
    mask_image = mask_image.resize((new_width, new_height))
    control_image = control_image[0].resize((new_width, new_height))

    prompt = "hand"
    strength=0.5
    controlnet_conditioning_scale = 1.0

    image = pipe(
        prompt=prompt,
        image=init_image,
        mask_image=mask_image,
        control_image=control_image,
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        strength=strength,
    ).images[0]

    image.save('result.jpg')
    
    
if __name__ == "__main__":
    inpaint_with_controlnet()

Feel free to add my discord and we can discuss there.

patrickvonplaten · 2023-08-25T17:01:05Z

Very cool PR! @yiyixuxu can you give this a look? :-)

yiyixuxu

Thanks! excellent work!

I think 2 main thing left are:

Refactor with a mask_image_processor https://github.com/huggingface/diffusers/pull/4444/files
Add MultiControlnet support

yiyixuxu · 2023-08-25T22:29:23Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+
+
+# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.prepare_mask_and_masked_image
+def prepare_mask_and_masked_image(image, mask, height, width, return_image=False):


We just deprecated this function :)
in this PR #4444 (comment)
let's update this PR too

Updated

self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor) self.mask_processor = VaeImageProcessor( vae_scale_factor=self.vae_scale_factor, do_normalize=False, do_binarize=True, do_convert_grayscale=True) self.control_image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True, do_normalize=False)

yiyixuxu · 2023-08-25T22:31:49Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+        self.control_image_processor = VaeImageProcessor(
+            vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True, do_normalize=False
+        )
+        self.watermark = StableDiffusionXLWatermarker()


add a mask_processor here

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

yiyixuxu · 2023-08-25T22:51:43Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+            generator = torch.Generator(device=device).manual_seed(seed)
+
+        controlnet_embedder_scale_factor = 2
+        control_image = randn_tensor(


I think we accept image tensor in [0,1] range, so should not use randn_tensor here

Thank you! Corrected.

control_image = ( floats_tensor( (1, 3, 32 * controlnet_embedder_scale_factor, 32 * controlnet_embedder_scale_factor), rng=random.Random(seed), ) .to(device) .cpu() )

yiyixuxu · 2023-08-25T22:56:02Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+        init_image = init_image.cpu().permute(0, 2, 3, 1)[0]
+
+        controlnet_embedder_scale_factor = 2
+        image = Image.fromarray(np.uint8(init_image)).convert("RGB").resize((64, 64))


the dummy image and mask_image are just 2 black images here

let's do something similar as https://github.com/huggingface/diffusers/pull/4536/files#diff-b65a24df736726ca6f92c71567b77c2a9832ee6142ee2dcbdb08e9addcb6da4b

Followed the link's code,

image = floats_tensor((1, 3, 32, 32), rng=random.Random(seed)).to(device) image = image.cpu().permute(0, 2, 3, 1)[0] mask_image = torch.ones_like(image) controlnet_embedder_scale_factor = 2 control_image = ( floats_tensor( (1, 3, 32 * controlnet_embedder_scale_factor, 32 * controlnet_embedder_scale_factor), rng=random.Random(seed), ) .to(device) .cpu() )

yiyixuxu · 2023-08-25T22:58:21Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+        assert np.abs(image_slice_1.flatten() - image_slice_3.flatten()).max() > 1e-4
+
+    # Ignore float16 for SDXL
+    def test_float16_inference(self):


why do we disable this?

This was unintentional. Removed the disabling.

kfzyqin · 2023-08-27T23:47:40Z

Thank you @yiyixuxu and @patrickvonplaten. I will work on comments this week.

kfzyqin · 2023-08-29T12:40:21Z

Borrowing ideas of PR 4811. Working in progress.

…sers into sdxl_ctrl_inpaint

patrickvonplaten · 2023-08-30T07:39:18Z

Hey @viiika,

Could we maybe work on this PR together? @harutatsuakiyama can you maybe invite @viiika as a collaborator for this PR to your fork so that we can work here?

@viiika , it's quite rare that we have two PRs about the same feature popping up almost at the same time - very sorry for the potentially duplicated work. Would it be ok to pass onto this PR because:

we already reviewed this PR
The PR was up a bit earlier

That would be very nice if we could collaborate here 🙏

patrickvonplaten · 2023-08-30T07:40:02Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+    return mask
+
+
+def prepare_mask_and_masked_image(image, mask, height, width, return_image: bool = False):


Can we remove this function and instead use the new mask processor logic: #4444

@harutatsuakiyama I think you can delete this function now if not used?

viiika · 2023-08-30T07:55:22Z

I still insist that #4811 already support some new features mentioned in #4694, like MultiControlnet, the api usage, no randn_tensor for control_image, even refactor with a mask_image_processor you mentioned just now, etc.

And the coding style is more consistent with pipeline_stable_diffusion_xl_inpaint, compared to StableDiffusionControlNetInpaintPipeline adapted from StableDiffusionInpaintPipeline.

I believe #4811 requires almost no effort to review, because it and the latest pipeline_stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint are updated synchronously.

Despite this, merge which PR depends you. And I believe if you choose #4811, it may take less than a day for us to merge.

viiika · 2023-08-30T08:02:16Z

Also, if you still insist we should continue with #4694, that's fine with me and I can try my best to help fixing problems. I just think merging #4694 will take a few weeks to handle many problems, and might introduce some design inconsistencies. A lot of current research relies on this pipeline, so I just hope it gets merged soon.

kfzyqin · 2023-09-01T10:54:10Z

Hi @yiyixuxu. Thanks for the review. I have addressed the review comments:

Update doc string.
Remove unnecessary functions.
Fix test errors.

My local tests show no issues. Please let me know if further changes are required :-)

patrickvonplaten · 2023-09-01T14:48:12Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+        ] = None,
+        height: Optional[int] = None,
+        width: Optional[int] = None,
+        strength: float = 1.0,


Suggested change

strength: float = 1.0,

strength: float =0.9999,

Changed, but why?

patrickvonplaten · 2023-09-01T14:48:29Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+                The height in pixels of the generated image.
+            width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor):
+                The width in pixels of the generated image.
+            strength (`float`, *optional*, defaults to 1.):


Suggested change

strength (`float`, *optional*, defaults to 1.):

strength (`float`, *optional*, defaults to 0.9999):

Changed, can I curiously ask why?

patrickvonplaten · 2023-09-01T14:49:24Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+
+            control_image = control_images
+        else:
+            assert False


Suggested change

assert False

raise ValueError(f"{controlnet.__class__} is not supported.")

patrickvonplaten

Good to merge once @yiyixuxu is ok with it :-)

patrickvonplaten · 2023-09-01T16:28:50Z

@viiika could you maybe drop your email here so that we can add you as a co-author via https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

viiika · 2023-09-01T16:36:35Z

@viiika could you maybe drop your email here so that we can add you as a co-author via https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

Sure. My primary GitHub email for this account is 1355864570@qq.com. Thank you very much!

yiyixuxu · 2023-09-01T16:50:20Z

@harutatsuakiyama
let's make sure the code quality checks pass. make style please :)

patrickvonplaten · 2023-09-01T20:52:27Z

@viiika could you maybe drop your email here so that we can add you as a co-author via https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

Sure. My primary GitHub email for this account is 1355864570@qq.com. Thank you very much!

@harutatsuakiyama could you add @viiika as an author here that would be very nice ❤️

Co-authored-by: Jiabin Bai 1355864570@qq.com

kfzyqin · 2023-09-02T01:43:07Z

Hi @yiyixuxu, @patrickvonplaten, and @viiika,

I have addressed the new code review comments:

Including @viiika as an author by including name and email in the commit
Change various number issues

For the failing tests, it seems previous failure was due to Internet issues (500 bad gate). My local tests can pass.

Please let me know if further changes are required.

yiyixuxu · 2023-09-02T03:11:10Z

@harutatsuakiyama
Could you run make fix-copies and make style -
Let's make sure CI is green

kfzyqin · 2023-09-02T03:41:37Z

Thank you @yiyixuxu. I just realized that diffusers.utils.dummy_torch_and_transformers_objects.py has some style problems. I have fixed them.

The following shows outputs of make fix-copies and make style. The errors of make style are not due to the code that I have uploaded. I think this time, the CI should be green :-)

Let me know if other things are required.

make fix-copies

python utils/check_copies.py --fix_and_overwrite
python utils/check_dummies.py --fix_and_overwrite

make style

black examples scripts src tests utils
All done! ✨ 🍰 ✨
613 files left unchanged.
ruff examples scripts src tests utils --fix
examples/community/lpw_stable_diffusion_xl.py:1141:42: E721 Do not compare types, use `isinstance()`
examples/community/stable_diffusion_xl_reference.py:703:42: E721 Do not compare types, use `isinstance()`
src/diffusers/experimental/rl/value_guided_sampling.py:79:12: E721 Do not compare types, use `isinstance()`
src/diffusers/pipelines/audio_diffusion/pipeline_audio_diffusion.py:181:12: E721 Do not compare types, use `isinstance()`
src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py:827:42: E721 Do not compare types, use `isinstance()`
src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py:909:20: E721 Do not compare types, use `isinstance()`
src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py:1132:20: E721 Do not compare types, use `isinstance()`
src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py:877:42: E721 Do not compare types, use `isinstance()`
tests/pipelines/consistency_models/test_consistency_models.py:190:12: E721 Do not compare types, use `isinstance()`
tests/pipelines/unidiffuser/test_unidiffuser.py:112:12: E721 Do not compare types, use `isinstance()`
tests/pipelines/unidiffuser/test_unidiffuser.py:548:12: E721 Do not compare types, use `isinstance()`
tests/pipelines/unidiffuser/test_unidiffuser.py:651:12: E721 Do not compare types, use `isinstance()`
Found 12 errors.
make: *** [Makefile:59: style] Error 1

kfzyqin · 2023-09-02T04:43:11Z

Ahh I see, I need to run the test for doc builder. Let me do that. I aim that to be the last test.

Sorry for failing test again. Can I ask for hints about how to fix this error? @yiyixuxu Also, can we get access to run tests, for more efficient debugging purposes? I have tried locally, and seem to be correct ...

All done! ✨ 🍰 ✨
617 files would be left unchanged.
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.17/x64/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/doc_builder/commands/style.py", line 28, in style_command
    raise ValueError(f"{len(changed)} files should be restyled!")
ValueError: 1 files should be restyled!
Error: Process completed with exit code 1.

yiyixuxu · 2023-09-02T04:38:56Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+        >>> mask_image = load_image(mask_url).convert("RGB")
+
+        >>> original_width, original_height = init_image.size
+        >>> new_width = int(original_width / 2)


why do we resize?

This is to save CUDA memory. Removed in the new code.

yiyixuxu · 2023-09-02T04:42:50Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+        self,
+        prompt: Union[str, List[str]] = None,
+        prompt_2: Optional[Union[str, List[str]]] = None,
+        image: Union[


let's use a custom type PipelineImageInput (was recently introduced)

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

Line 891 in 5eeedd9

image: PipelineImageInput = None,

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

yiyixuxu · 2023-09-02T04:44:46Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+            List[PIL.Image.Image],
+            List[np.ndarray],
+        ] = None,
+        mask_image: Union[torch.FloatTensor, PIL.Image.Image] = None,


I think mask_image should be of same type as image no? PipelineImageInput

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

yiyixuxu · 2023-09-02T04:53:34Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

+                    latent_model_input = torch.cat([latent_model_input, mask, masked_image_latents], dim=1)
+
+                # predict the noise residual
+                added_cond_kwargs = {"text_embeds": add_text_embeds, "time_ids": add_time_ids}


I don't think this line is needed? it has not changed from line 1452

yiyixuxu · 2023-09-02T04:54:29Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+            projection_class_embeddings_input_dim=80,  # 6 * 8 + 32
+            cross_attention_dim=64,
+        )
+        torch.manual_seed(0)


Why do we need to fix the seed here? I don't think we have any randomness here, no?

I followed the test here: https://github.com/huggingface/diffusers/blob/main/tests/pipelines/controlnet/test_controlnet_sdxl.py

yiyixuxu · 2023-09-02T04:56:20Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+    image_latents_params = TEXT_TO_IMAGE_IMAGE_PARAMS
+
+    def get_dummy_components(self):
+        torch.manual_seed(0)


is this needed?

Follow test here: https://github.com/huggingface/diffusers/blob/main/tests/pipelines/controlnet/test_controlnet_sdxl.py

yiyixuxu · 2023-09-02T04:56:31Z

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

+            projection_class_embeddings_input_dim=80,  # 6 * 8 + 32
+            cross_attention_dim=64,
+        )
+        torch.manual_seed(0)


same, needed?

Similarly, follow test here: https://github.com/huggingface/diffusers/blob/main/tests/pipelines/controlnet/test_controlnet_sdxl.py

tests/pipelines/controlnet/test_controlnet_inpaint_sdxl.py

yiyixuxu · 2023-09-02T05:11:59Z

regards to the quality test, make sure you are up to date? pip install --upgrade -e .["quality"]

cc @DN6 here we need help with tests!

kfzyqin · 2023-09-02T05:21:07Z

I found out the test issues, some lines in doc_string is too long.

…tting errors

kfzyqin · 2023-09-02T05:43:29Z

Hi @yiyixuxu. I removed EXAMPLE_DOC_STRING since it keeps getting errors for doc-builder style src/diffusers docs/source --max_len 119 --check_only --path_to_docs docs/source. In the future, I will try getting it back, maybe need some help from the test experts :-)

For now, I strongly believe the code should be able to pass tests (finger crossed 🙏)

…ss_mode

kfzyqin · 2023-09-02T07:03:48Z

Hi @yiyixuxu, thanks for the new review round. I have addressed the comments:

Code now uses PipelineImageInput.
Add guess_mode.
Add EXAMPLE_DOC_STRING.
Add test for guess_mode.

Also, I strongly believe the code should be able to pass tests (finger crossed 🙏)

Let me know if further changes are required.

…uggingface#4694) * [ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL Co-authored-by: Jiabin Bai 1355864570@qq.com --------- Co-authored-by: Harutatsu Akiyama <kf.zy.qin@gmail.com>

kfzyqin and others added 3 commits August 21, 2023 20:58

[ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL

9e718b6

[ControlNet SDXL Inpainting] Modify __init__.py for importing

fa41ede

Merge branch 'main' into sdxl_ctrl_inpaint

0d743bc

kfzyqin marked this pull request as draft August 21, 2023 21:59

kfzyqin changed the title ~~[ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL~~ [(Draft) ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL Aug 21, 2023

kfzyqin added 2 commits August 22, 2023 21:51

controlnet_inpainter_sdxl.py

050e19d

Merge branch 'sdxl_ctrl_inpaint' of github.com:harutatsuakiyama/diffu…

d66556f

…sers into sdxl_ctrl_inpaint

kfzyqin marked this pull request as ready for review August 22, 2023 11:53

kfzyqin changed the title ~~[(Draft) ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL~~ [ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL Aug 22, 2023

yiyixuxu reviewed Aug 25, 2023

View reviewed changes

yiyixuxu mentioned this pull request Aug 28, 2023

Add StableDiffusionXLControlNetInpaintPipeline #4811

Closed

6 tasks

kfzyqin and others added 3 commits August 29, 2023 22:30

[ControlNet SDXL Inpainting] Update pipeline_controlnet_inpaint_sd_xl.py

dfa8ab2

[ControlNet SDXL Inpainting] Update pipeline_controlnet_inpaint_sd_xl.py

bba627c

Merge branch 'main' into sdxl_ctrl_inpaint

b221ec4

kfzyqin added 3 commits August 30, 2023 00:07

[ControlNet SDXL Inpainting] Update pipeline_controlnet_inpaint_sd_xl.py

593da7e

Merge branch 'sdxl_ctrl_inpaint' of github.com:harutatsuakiyama/diffu…

01d9766

…sers into sdxl_ctrl_inpaint

[ControlNet SDXL Inpainting] Update pipeline_controlnet_inpaint_sd_xl.py

73e2699

patrickvonplaten reviewed Aug 30, 2023

View reviewed changes

[ControlNet SDXL Inpainting] Support MultiControlNet

62dd407

kfzyqin added 3 commits September 1, 2023 20:17

[ControlNet SDXL Inpainting] Fix __init__ style

9e1a51c

[ControlNet SDXL Inpainting] Update example doc string

be3dfb4

[ControlNet SDXL Inpainting] Remove unused functions

fba6757

patrickvonplaten reviewed Sep 1, 2023

View reviewed changes

patrickvonplaten approved these changes Sep 1, 2023

View reviewed changes

[ControlNet SDXL Inpainting] Address code review;

7ebc62f

Co-authored-by: Jiabin Bai 1355864570@qq.com

[ControlNet SDXL Inpainting] Fix dummy style

a6e37ba

yiyixuxu reviewed Sep 2, 2023

View reviewed changes

[ControlNet SDXL Inpainting] Remove EXAMPLE_DOC_STRING as it keeps ge…

ccf25a7

…tting errors

kfzyqin added 2 commits September 2, 2023 17:00

[ControlNet SDXL Inpainting] Add EXAMPLE_DOC_STRING back; Support gue…

e7fdce4

…ss_mode

[ControlNet SDXL Inpainting]Add test for guess_mode

5f4ecb0

yiyixuxu approved these changes Sep 2, 2023

View reviewed changes

yiyixuxu merged commit c52acaa into huggingface:main Sep 2, 2023



		# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.prepare_mask_and_masked_image
		def prepare_mask_and_masked_image(image, mask, height, width, return_image=False):

		return mask


		def prepare_mask_and_masked_image(image, mask, height, width, return_image: bool = False):

	strength (`float`, optional, defaults to 1.):
	strength (`float`, optional, defaults to 0.9999):

	assert False
	raise ValueError(f"{controlnet.__class__} is not supported.")

[ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL #4694

[ControlNet SDXL Inpainting] Support inpainting of ControlNet SDXL #4694

Uh oh!

Conversation

kfzyqin commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview:

Files Modified/Added:

Visualizations:

Overview:

Files Modified/Added:

Example Usage

Features

Uh oh!

Cathy0908 commented Aug 22, 2023

Uh oh!

kfzyqin commented Aug 22, 2023

Uh oh!

kfzyqin commented Aug 22, 2023

Uh oh!

Cathy0908 commented Aug 23, 2023

Uh oh!

kfzyqin commented Aug 23, 2023

Uh oh!

patrickvonplaten commented Aug 25, 2023

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfzyqin Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfzyqin commented Aug 27, 2023

Uh oh!

kfzyqin commented Aug 29, 2023

Uh oh!

patrickvonplaten commented Aug 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viiika commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viiika commented Aug 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kfzyqin commented Sep 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kfzyqin commented Aug 21, 2023 •

edited

Loading

kfzyqin Aug 31, 2023 •

edited

Loading

viiika commented Aug 30, 2023 •

edited

Loading

viiika commented Aug 30, 2023 •

edited

Loading

kfzyqin commented Sep 1, 2023 •

edited

Loading

kfzyqin commented Sep 2, 2023 •

edited

Loading

kfzyqin Sep 2, 2023 •

edited

Loading