Outputs of Idefics2 is unrelated to the images in the latest versions of Transformers

[HugoLaurencon]

System Info

• transformers version: 4.43.2
• Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.17
• Python version: 3.8.18
• Huggingface_hub version: 0.23.2
• Safetensors version: 0.4.1
• Accelerate version: 0.26.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu118 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:

Who can help?

@amyeroberts
@ArthurZucker

Just filling an official issue so that other users can see it.
Some people didn't understand why they had much lower scores with Idefics2, as it's a silent bug, plus the model still gives generations related to the text prompt, so that's hard to notice at first there is a bug.
I also put it in big in the model cards like https://huggingface.co/HuggingFaceM4/idefics2-8b

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq


processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/idefics2-8b", torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="flash_attention_2"
)


# Create inputs
path_image = "/fsx/hugo/wow_images/cv_0.png"
image = Image.open(path_image)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What's written on this image?"},
        ]
    },      
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_texts)

Pick any image, this code will give different outputs, depending on the version of Transformers.
It gives correct outputs for version 4.40, and outputs that seem unrelated to the image with newer versions.
This is surely because of the new caching strategy implemented in Transformers, and modeling_idefics2.py doesn't that take into account.

Expected behavior

We should have the same output as with the version 4.40 with the newer versions.

[amyeroberts]

Thanks for reporting this @HugoLaurencon!

I've dug into this more, and the issue is to do with the flash attention call in the perceiver layer, rather than the cache.

I don't have access to the image in the issue description and my machine isn't large enough to test with more than one sample and image unfortunately, so tested with the following script. If this isn't properly capturing the behaviour being reported, let me know!

import requests
import torch
from PIL import Image
from io import BytesIO

import transformers
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda"

image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")

attn_implementation = "eager"
# attn_implementation = "flash_attention_2"

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation=attn_implementation,
)

# Create inputs
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)

print("transformers version:", transformers.__version__)
print("attn_implementation:", attn_implementation)
print("\nPrompt:")
print(prompt)

inputs = processor(text=prompt, images=[image1], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}

# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print("\nGenerated text:")
print(generated_texts[0])

Tested image for reference:

v4.40.2

From the issue report, if we run on v4.40 (last known working release) we get the following:

Eager

transformers version: 4.40.2
attn_implementation: eager

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:
The `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead.

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a statue, there are some buildings, trees, also we can see the sky.

Flash Attention

transformers version: 4.40.2
attn_implementation: flash_attention_2

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:
The `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead.

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a statue, there are some buildings, trees, also we can see the sky.

We can see that the generated outputs are the same.

main

And, as reported, running on main these differ and the output when using flash attention isn't related to the image at all

Eager

transformers version: 4.44.0.dev0
attn_implementation: eager

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a statue, there are some buildings, trees, also we can see the sky.

Flash Attention

transformers version: 4.44.0.dev0
attn_implementation: flash_attention_2

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a poster. On the poster we can see a person and some text.

v4.42.0

The model appears to still generate sensible outputs after v4.42

transformers version: 4.42.0
attn_implementation: flash_attention_2

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a statue, there are some buildings, trees, also we can see the sky.

#32275

Running on the branch with the fix we get the same as eager again :)

transformers version: 4.44.0.dev0
attn_implementation: flash_attention_2

Prompt:
User:<image>What do we see in this image?<end_of_utterance>
Assistant:

Generated text:
User: What do we see in this image? 
Assistant: In this image we can see a statue, there are some buildings, trees, also we can see the sky.

[HugoLaurencon]

Thanks a ton Amy, I confirm I obtain the same outputs now!