Update default values of bos/eos token ids in CLIPTextConfig

[ydshieh]

What does this PR do?

Currently the default values are not the ones from the corresponding tokenizers.

See discussion in #24650

However, we can't use the config.eos_token_id in the modeling file (which is the ultimate goal in #24650) with only the change in this PR. We will have to update all the Hub repo. config files first 😢 . (Probably there is something easier to do)

[ydshieh]

This PR just updates the default values. For existing Hub repo. config files, their values are still used.

[ydshieh]

I will remove this TODO block once PR being reviewed.

[ydshieh]

When I use the tokenizer from the openai/clip repository, the padding is done with 0. But in HF CLIP tokenizer, it is eos_token_id, and the config here has default value 1.

Although the padding is not used for pooling in clip text model, and should not affect the text-image similarity loss computation, I am afraid to change it here.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

Regarding the padding token:

(copy past from (partial) internal discussion given @patil-suraj)

When we added CLIP I tested for the text_projection , logits_per_image and logits_per_text. For the text_projection the model pulls the embeddings of the last token i.e the eos token. The rest of the tokens i.e the padding tokens are ignored. We can see in this colab that text_projection , logits_per_image and logits_per_text match with the OAI model because we only take the pooled embeddings. And when CLIP was released it was intended for these features which are needed for contrastive tasks. Hence I didn't test against all token embeddings.

IMO the wrong padding token will only affect inference when using all token emebeddings i.e Stable Diffusion. For training even if the padding token is wrong it shouldn't affect because

• Because CLIP did not use attention_mask during training.

• CLIPTextEncoder uses casual mask, so the tokens to the right don't influence the hidden states of tokens to the left.
• CLIP is trained with contrastive loss which is computed using the projections, and as I said above the text_projection is computed by pooling the eos token embeddings, which will be always similar no matter what the padding token is, because CLIPTextEncoder is causal, so the eos embeddings won't be affected by tokens on the right.
• Hence, for downstream training (like SD) as long as a consistent token is used for padding it shouldn't severely affect the training. But for inference we will need to use the same token as Patrick explained.
  This could also be the reason that we didn't have any issue related to this.

As far as I can understand, it'll only affect the inference if a different token (compared to the padding token used for training) is used for padding. (edited)

[amyeroberts]

Thanks for updating, and the detailed comments & explanations! 🤗