Support MetaCLIP 2

[NielsRogge]

What does this PR do?

Meta just released MetaCLIP 2 (worldwide), new CLIP models trained on 300+ languages.

However, when making them compatible with modeling_clip.py, I noticed there's a mistake with the original OpenAI CLIP models.

• they have the EOS token ID set to 2 in the config: https://huggingface.co/openai/clip-vit-large-patch14/blob/main/config.json#L25. However, the OpenAI CLIP models don't use 2 as EOS token ID. They use 49407. You can check this when tokenizing text using CLIPTokenizer:

>>> from transformers import CLIPTokenizer
>>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
>>> input_ids = tokenizer("hello world", return_tensors="pt")
>>> input_ids
{'input_ids': tensor([[49406,  3306,  1002, 49407]]), 'attention_mask': tensor([[1, 1, 1, 1]])}

• this was fixed in Update default values of bos/eos token ids in CLIPTextConfig #24773. For backwards compatibility, a block of code was kept and it only runs in case a model has wrongly set EOS token ID == 2.
• since MetaCLIP 2 actually uses EOS token ID == 2 with a multilingual tokenizer (https://huggingface.co/facebook/xlm-v-base), it needs the "else" block which gets the EOS token from each sequence along the batch dimension.
• this means we'd need to adapt the "if" block. I propose here to simply check whether the max values in each row of the input_ids corresponds to 49407, the value all OpenAI CLIP models use.

cc @ydshieh

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: clip

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

can't be 100% sure if there are clip models on the hub with self.eos_token_id == 2 and yet with tokenizer's eos != 49407 or if already having added new tokens.

[ArthurZucker]

also 49407 is hardcoded

[zucchini-nlp]

should we start raising warning and deprecate an incorrect 'eos', so this code bloack can be removed in the future? Super hacky rn

[NielsRogge]

That would require updating the configs of the OpenAI CLIP models, models like FashionCLIP, BioCLIP, etc (probably thousands of models)

[ydshieh]

As talked internally with Arthur, let's not touch this part of code, just simply use modular.

@NielsRogge suggest to run some tests to avoid this kind of situation in the future.