Flux does not support multiple Controlnets?

[RimoChan]

Describe the bug

I'm encountering an issue with the FluxControlNetPipeline. The controlnet parameter is supposed to accept a List[FluxControlNetModel]. However, when I attempt to execute my code, I run into the following error:

Traceback (most recent call last):
  File "/opt/tiger/test_1/h.py", line 8, in <module>
    pipe = FluxControlNetPipeline.from_pretrained('/mnt/bn/x/sd_models/flux_schnell/', controlnet=controlnet, torch_dtype=torch.bfloat16).to("cuda")
  File "/opt/tiger/miniconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/tiger/miniconda3/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 940, in from_pretrained
    model = pipeline_class(**init_kwargs)
  File "/opt/tiger/miniconda3/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux_controlnet.py", line 206, in __init__
    self.register_modules(
  File "/opt/tiger/miniconda3/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 162, in register_modules
    library, class_name = _fetch_class_library_tuple(module)
  File "/opt/tiger/miniconda3/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 731, in _fetch_class_library_tuple
    library = not_compiled_module.__module__.split(".")[0]
AttributeError: 'list' object has no attribute '__module__'. Did you mean: '__mul__'?

Reproduction

import torch
from diffusers import FluxControlNetPipeline, FluxControlNetModel

controlnet = [
    FluxControlNetModel.from_pretrained("InstantX/FLUX.1-dev-controlnet-canny", torch_dtype=torch.bfloat16),
    FluxControlNetModel.from_pretrained("InstantX/FLUX.1-dev-controlnet-canny", torch_dtype=torch.bfloat16),
]
pipe = FluxControlNetPipeline.from_pretrained('/mnt/bn/x/sd_models/flux_schnell/', controlnet=controlnet, torch_dtype=torch.bfloat16).to("cuda")

Logs

No response

System Info

• 🤗 Diffusers version: 0.31.0.dev0
• Platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
• Running on Google Colab?: No
• Python version: 3.10.14
• PyTorch version (GPU?): 2.3.1+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.24.5
• Transformers version: 4.38.2
• Accelerate version: 0.33.0
• PEFT version: 0.12.0
• Bitsandbytes version: 0.44.1
• Safetensors version: 0.4.4
• xFormers version: 0.0.27
• Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[hlky]

Linked PR adds the missing conversion from list/tuple to FluxMultiControlNetModel, in the meantime you can pass in FluxMultiControlNetModel yourself:

import torch
from diffusers import FluxControlNetPipeline, FluxControlNetModel, FluxMultiControlNetModel

controlnet = FluxMultiControlNetModel([
    FluxControlNetModel.from_pretrained("InstantX/FLUX.1-dev-controlnet-canny", torch_dtype=torch.bfloat16),
    FluxControlNetModel.from_pretrained("InstantX/FLUX.1-dev-controlnet-canny", torch_dtype=torch.bfloat16),
])
pipe = FluxControlNetPipeline.from_pretrained('/mnt/bn/x/sd_models/flux_schnell/', controlnet=controlnet, torch_dtype=torch.bfloat16).to("cuda")