Skip to content

Embedding VLMs don't need a head#45000

Merged
zucchini-nlp merged 5 commits intohuggingface:mainfrom
zucchini-nlp:col-models
Mar 27, 2026
Merged

Embedding VLMs don't need a head#45000
zucchini-nlp merged 5 commits intohuggingface:mainfrom
zucchini-nlp:col-models

Conversation

@zucchini-nlp
Copy link
Copy Markdown
Member

@zucchini-nlp zucchini-nlp commented Mar 25, 2026 •
edited
Loading

What does this PR do?

As per title, after #44976 users will be seeing a missing_weights - lm_head not found error even though the model doesn't use an lm head

On the way also deleted unnecessary methods, which is same as base class

@zucchini-nlp
Copy link
Copy Markdown
Member Author

zucchini-nlp commented Mar 25, 2026

run_slow: colpali, colqwen2

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 25, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/colpali", "models/colqwen2"]
quantizations: []

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Mar 25, 2026

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 25, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN c4ce8d8e workflow commit (merge commit)
PR 639b5622 branch commit (from PR)
main 2f624917 base commit (on main)

Model CI Report

❌ 1 new failed tests from this PR 😭

  • colpali:
    tests/models/colpali/test_modeling_colpali.py::ColPaliForRetrievalModelTest::test_correct_missing_keys (✅ ⟹ ❌)

ArthurZucker
ArthurZucker approved these changes Mar 26, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good!

@zucchini-nlp
Copy link
Copy Markdown
Member Author

zucchini-nlp commented Mar 26, 2026

run-slow: colmodernvbert, colpali, colqwen2

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 26, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/colmodernvbert", "models/colpali", "models/colqwen2"]
quantizations: []

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 26, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 539b94d5 workflow commit (merge commit)
PR 17603f3d branch commit (from PR)
main 09832b2a base commit (on main)

Model CI Report

❌ 7 new failed tests from this PR 😭

  • colqwen2:
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_bc_torch_dtype (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_can_use_safetensors (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_cpu_offload (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_disk_offload_bin (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_disk_offload_safetensors (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)
    tests/models/colqwen2/test_modeling_colqwen2.py::ColQwen2ForRetrievalModelTest::test_save_load (✅ ⟹ ❌)

@zucchini-nlp
Copy link
Copy Markdown
Member Author

zucchini-nlp commented Mar 27, 2026

run-slow: colqwen2

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 27, 2026

[For maintainers] Suggested jobs to run (before merge)

run-slow: colmodernvbert, colpali, colqwen2

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 27, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/colqwen2"]
quantizations: []

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 27, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 14158e90 workflow commit (merge commit)
PR 820bab1e branch commit (from PR)
main 23773e71 base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

@zucchini-nlp zucchini-nlp added this pull request to the merge queue Mar 27, 2026
Merged via the queue into huggingface:main with commit 4ee7f51 Mar 27, 2026
30 checks passed
@zucchini-nlp zucchini-nlp deleted the col-models branch March 27, 2026 09:57
zucchini-nlp added a commit to zucchini-nlp/transformers that referenced this pull request Mar 27, 2026
@zucchini-nlp
Embedding VLMs don't need a head (huggingface#45000)
49d17c7 
* squash

* fix copies

* skip, we dont need to load base model for it

* oops, one more regex since now we have no prefix
NielsRogge pushed a commit to NielsRogge/transformers that referenced this pull request Mar 30, 2026
@zucchini-nlp @NielsRogge
Embedding VLMs don't need a head (huggingface#45000)
5e0c80c 
* squash

* fix copies

* skip, we dont need to load base model for it

* oops, one more regex since now we have no prefix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ArthurZucker ArthurZucker ArthurZucker approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zucchini-nlp @HuggingFaceDocBuilderDev @ArthurZucker