StableDiffusionSafetyChecker ignores `attn_implementation` load kwarg

[jambayk]

Describe the bug

transformers added sdpa and FA2 for CLIP model in huggingface/transformers#31940. It now initializes the vision model like https://github.com/huggingface/transformers/blob/85a1269e19af022e04bc2aad82572cd5a9e8cdd9/src/transformers/models/clip/modeling_clip.py#L1143.

However, StableDiffusionSafetyChecker uses

diffusers/src/diffusers/pipelines/stable_diffusion/safety_checker.py

Line 41 in 2c25b98

self.vision_model = CLIPVisionModel(config.vision_config)

so it always gets initialized with sdpa attention.

Reproduction

from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker

model = StableDiffusionSafetyChecker.from_pretrained(
    "runwayml/stable-diffusion-v1-5", 
    subfolder="safety_checker", 
   attn_implementation="eager"
)
print(type(model.vision_model.vision_model.encoder.layers[0].self_attn))

Expected transformers.models.clip.modeling_clip.CLIPAttention but got transformers.models.clip.modeling_clip.CLIPSdpaAttention.

Logs

No response

System Info

diffusers 0.29.0
transformers 4.43.1

Who can help?

@sayakpaul @dn

[sayakpaul]

Thanks for bringing this up. CLIP FA2 and SDPA support is very recent. If you want to open a PR to fix this, we are happy to guide you.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Gentle ping @jambayk if you would like to help contribute support for this!

[jambayk]

Hi @a-r-r-o-w, sorry for missing the previous reply about contributing. Unfortunately, I don't think I can make the contribution for this as we just ended up pinning the diffuser version in our example.

[a-r-r-o-w]

Oh okay, we'll see how we can try improving this. Sorry for the delay! cc @sayakpaul

[sayakpaul]

SDPA for CLIP is in transformers for a while. Could you please try updating your installation of transformers retry?