KeyError: 'pi'

[zhousteven]

System enviroment:
Ubuntu20.04
torch 2.0.0+cu118
torchvision 0.15.1+cu118

Commandline:
#run_vqav2_ft.py --train_config_file=vqa_train_config.json

Error Description as below:
/home/steven/anaconda3/envs/nlp/bin/python /home/steven/workstore/nlp/VLE-main/run_vqav2_ft.py --train_config_file=vqa_train_config.json
/home/steven/workstore/nlp/VLE-main/run_vqav2_ft.py:76: SyntaxWarning: "is" with a literal. Did you mean "=="?
max_epochs=_config["max_epoch"] if max_steps is -1 else 1000,
2023-09-20 13:52:41.071596: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Global seed set to 0
Some weights of VLEForVQA were not initialized from the model checkpoint at hfl/vle-base and are newly initialized: ['vqa_classifier.1.bias', 'vqa_classifier.3.bias', 'vqa_classifier.3.weight', 'vqa_classifier.0.bias', 'vqa_classifier.1.weight', 'vqa_classifier.0.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
File "/home/steven/workstore/nlp/VLE-main/run_vqav2_ft.py", line 107, in
main(train_config)
File "/home/steven/workstore/nlp/VLE-main/run_vqav2_ft.py", line 25, in main
model = VLEForVQA_PL(_config)
File "/home/steven/workstore/nlp/VLE-main/vqav2_train_module.py", line 73, in init
new_state_dict = extend_position_embedding(self.model.state_dict(), patch_size, config["image_size"])
File "/home/steven/workstore/nlp/VLE-main/models/VLE/modeling_vle.py", line 124, in extend_position_embedding
state_dict[keys['pi'][0]] = torch.arange(grid_after*grid_after + 1).unsqueeze(0)
KeyError: 'pi'

Process finished with exit code 1

when i debug step by step, i found that cannot found key value "vision_model.embeddings.position_ids" in parameters list of model 'vle-base'.
Any body encounter same question as me?
please kindly help to solve this problem!
Tks~

[zhousteven]

@ymcui @GoGoJoestar @airaria waiting for your response

[GoGoJoestar]

The pi was added to keys if there is a parameter ends with vision_model.embeddings.position_ids. The related parameter model.vle.vision_model.vision_model.embeddings.position_ids is a buffer registered in the class transformers.models.clip.modeling_clip.CLIPVisionEmbeddings, you can check if it in the model vle-base as following:

for n,b in model.named_buffers():
    print(n, b.shape)

If it not in the model, you could check the related code of CLIP model in transformers, or try reinstalling transformers, re-pull VLE repo.