CLIP

[patil-suraj]

What does this PR do?

This PR adds the CLIP model.

CLIP is a multi-modal vision+language model which uses a transformer model for encoding both the images and text.

• The model here is designed such that both CLIPTextModel and CLIPVisionModel can be loaded independently, and composed together to get the CLIPModel.
• Both CLIPTextModel and CLIPVisionModel use the shared encoder class CLIPEncoder.
• The config classes are also kept in separate i.e CLIPTextConfig and CLIPVisionConfig. This could be kept in one config class but then we would have to add two arguments for each config value i.e text_hidden_size for text model vision_hidden_size for vision model etc.
  One issue here is that when we load an individual model, like CLIPTextModel using the weights of the whole CLIPModel
  the config ends up containing both text and vision config dicts, this does not cause any issue but could be confusing to look at.

One important thing to note here is that CLIP's tokenizer does have a pad token defined for it, but they use 0 as pad_token_id to pad the text, but the token, but the token associated with 0 is not a pad token. So here, to able to do padding I've added pad_token_id as a property which returns 0. I would be happy to hear if there is some other way to achieve this.

Also, I've added a processor class here but not sure if we really need it for this model. We could easily use the extractor for the vision model and tokenizer for the text model.

Would love your review about the design @LysandreJik , @patrickvonplaten , @sgugger.

[patil-suraj]

All green!!
I've addressed most of the suggestions, notably

• new processor API => as discussed with @patrickvonplaten and @LysandreJik processor's __call__ now accepts both the text and/or images and returns a single encoding dict. as_target_processor is now removed. The API is as follows

model = CLIPModel.from_pretrained(checkpoint)
inputs = CLIPProcessor(texts=..., images=..., some_other_kwargs)
outputs = model(**inputs)

• the encode_text and encode_image methods are renamed to get_text_features and get_image_features
• Added fast tokenizer.

Ready for second review @LysandreJik @sgugger @patrickvonplaten

[sgugger]

Great work! A few last loose ends to tie up (in particular don't forget to replace all checkpoint names in the docstrings by ones in the openai namespace!) and it should be good to merge.

[sgugger]

Still standing :-)

[LysandreJik]

Great, LGTM! Only need to update from valhalla namespace to openai namespace and it looks good to me.

[lycfight]

All green!!
I've addressed most of the suggestions, notably

• new processor API => as discussed with @patrickvonplaten and @LysandreJik processor's __call__ now accepts both the text and/or images and returns a single encoding dict. as_target_processor is now removed. The API is as follows

model = CLIPModel.from_pretrained(checkpoint)
inputs = CLIPProcessor(texts=..., images=..., some_other_kwargs)
outputs = model(**inputs)

• the encode_text and encode_image methods are renamed to get_text_features and get_image_features
• Added fast tokenizer.

Ready for second review @LysandreJik @sgugger @patrickvonplaten

How to use processor in getitem()? I got an error"RuntimeError: stack expects each tensor to be equal size, but got [1, 11] at entry 0 and [1, 13] at entry 1" ,as follow:
def getitem(self, idx):
img_id = self.img_ids[idx]
# randomly pick one caption from the image captions
text = random.choice(self.img_id_to_captions[img_id])
img_filename = self.img_id_to_filename[img_id]
img_path = op.join(self.img_dir, img_filename)
img = Image.open(img_path)
input = self.processor(text = text, images = img, return_tensors = "pt", padding = True)
return input
I thought processor might need other args, inherited from pretraintokenizerbase,such as padding.But I couldn't find it at processor's call in doc.

[patil-suraj]

Hi @lycfight could you please open an issue with a minimal code snippet so we could take a look. Thanks :)

[lycfight]

Hi @lycfight could you please open an issue with a minimal code snippet so we could take a look. Thanks :)

of course