Skip to content

Ignore CLIP position_ids in unexpected key loading report#45385

Closed
songyuc wants to merge 1 commit intohuggingface:mainfrom
songyuc:fix/clip-ignore-position-ids-unexpected-keys
Closed

Ignore CLIP position_ids in unexpected key loading report#45385
songyuc wants to merge 1 commit intohuggingface:mainfrom
songyuc:fix/clip-ignore-position-ids-unexpected-keys

Conversation

@songyuc
Copy link
Copy Markdown

@songyuc songyuc commented Apr 12, 2026

What does this PR do?

Loading openai/clip-vit-base-patch32 currently reports the following keys as unexpected:

  • text_model.embeddings.position_ids
  • vision_model.embeddings.position_ids

In the current CLIP implementation, these buffers are registered with persistent=False, so they are not part of the current serialization contract and are not expected to be restored from checkpoints.

Older CLIP checkpoints may still contain these keys, but they are harmless historical buffers. This PR adds them to _keys_to_ignore_on_load_unexpected in CLIPPreTrainedModel so they no longer appear as misleading unexpected keys during loading.

This follows the same approach as #44508 for OwlViT.

Related to #44493.

I reproduced the issue locally, verified that the two keys disappear from unexpected_keys after the change, and added a targeted regression test.

Code Agent Policy

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Test plan

  • Reproduced the issue locally with:
    • CLIPModel.from_pretrained("openai/clip-vit-base-patch32", output_loading_info=True)
  • Added a slow test covering the loading report
  • Ran:
    • RUN_SLOW=1 python -m pytest tests/models/clip/test_modeling_clip.py -k position_ids -sv -rs

Who can review?

@Cyrilvallez

@github-actions
Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: clip

@songyuc
Copy link
Copy Markdown
Author

songyuc commented Apr 18, 2026

Thanks! This PR appears to be superseded by #45437, which has now been merged and also covers the CLIP case. I'll close this PR to avoid duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant