UNet2DCondition model can be passed `attention_head_dim` of wrong length

[williamberman]

Describe the bug

When passed attention_head_dim of wrong length, it will silently be used. The encoder will start indexing from the front of the list and the decoder will start indexing from the back.

Reproduction

This test is an example of an incorrectly parameterized conditional unet. It has two blocks each in the encoder/decoder but 4 different attention_head_dim's

diffusers/tests/pipelines/stable_diffusion_2/test_stable_diffusion.py

Lines 49 to 61 in 8178c84

	unet = UNet2DConditionModel(
	block_out_channels=(32, 64),
	layers_per_block=2,
	sample_size=32,
	in_channels=4,
	out_channels=4,
	down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
	up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
	cross_attention_dim=32,
	# SD2-specific config below
	attention_head_dim=(2, 4, 8, 8),
	use_linear_projection=True,
	)

Logs

No response

System Info

n/a

[patrickvonplaten]

Thanks for the issue! Think we should just raise a nice error in this case no?

Would you like to open a PR for this?