Can we have control mode for controlnet?

[alexblattner]

Is your feature request related to a problem? Please describe.
I'd like more options for controlnet usage

Describe the solution you'd like
implement this: Mikubill/sd-webui-controlnet#1011

Describe alternatives you've considered
Implementing it myself

Additional context
Add any other context or screenshots about the feature request here.

[alexblattner]

I'm gonna bump

[patrickvonplaten]

@alexblattner, please don't bump every issue after just one day, we're drowning in pings and this is getting a bit too much.

Also it would be great to change the feature request a bit from "Implement this" to something more specific. We would need some pseudo-code, design considerations etc... here.

How would you like to have "more control" implemented? An input to the call ? Do we even have to change core diffusers here?

[alexblattner]

@patrickvonplaten

An input to the call ?

Yes, I think having a parameter passed named control_mode with one of 3 values: "balanced", "prompt", "controlnet" the latter 2 are shortened versions of prompt more important and controlnet more important.

As for the old code that uses guess_mode, I think it's fine to keep it there for people who are already using this in their code. If both guess_mode and control_mode are enabled, the latter trumps it.

How would you like to have "more control" implemented?

As for implementing this in diffusers (it's called control mode but it's nothing too fancy from what I can tell), I have no idea. However, illyasiev would be glad to work with your team on it I am sure. In general, I think you would have an easier time if you reached out to external developers of great new features such as illyasiev to help you bring their code up to date.

Do we even have to change core diffusers here?

Only controlnet pipelines.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yan-Ops]

@alexblattner Has the problem been resolved? I have also encountered this problem. If there is a solution, can you share it?

[alexblattner]

@yan-Ops not yet :(