Skip to content

[Model] Update Molmo Eval to Match Official Implementation#648

Merged
kennymckormick merged 2 commits intoopen-compass:mainfrom
jamespark3922:release/molmo_prompt
Dec 30, 2024
Merged

[Model] Update Molmo Eval to Match Official Implementation#648
kennymckormick merged 2 commits intoopen-compass:mainfrom
jamespark3922:release/molmo_prompt

Conversation

@jamespark3922
Copy link
Copy Markdown
Contributor

@jamespark3922 jamespark3922 commented Dec 5, 2024

Changes:

  • Process up to max 36 crops per image instead of 12 crops in default setting.
  • Integrate prompts from the official Molmo implementation to match the reported numbers in the paper. (https://molmo.allenai.org/paper.pdf).

@jamespark3922
Copy link
Copy Markdown
Contributor Author

jamespark3922 commented Dec 13, 2024

@kennymckormick Could you take a look at this PR?

@kennymckormick
Copy link
Copy Markdown
Member

kennymckormick commented Dec 19, 2024

Hi, @jamespark3922 ,
Will merge this PR after validating the implementation with 1 model.

@kennymckormick
Copy link
Copy Markdown
Member

kennymckormick commented Dec 30, 2024

  • TODO: Re-evaluate Molmo Series

@kennymckormick kennymckormick merged commit c08ab64 into open-compass:main Dec 30, 2024
@kennymckormick
Copy link
Copy Markdown
Member

kennymckormick commented Jan 1, 2025

Hi, @jamespark3922 ,
I find that when using this piece of codes for evaluation, the scores of Molmo-7B-O and Molmo-7B-D on MMVet dropped by around 10% (absolute). Would you please evaluate these two models on MMVet to double check?

kennymckormick pushed a commit to TobiasLee/VLMEvalKit that referenced this pull request Jan 1, 2025
@jamespark3922 @kennymckormick
[Model] Update Molmo Eval to Match Official Implementation (open-comp…
40bbc75 
…ass#648)

* add molmo prompts

* fix lint format
kennymckormick added a commit that referenced this pull request Jan 1, 2025
@TobiasLee @kennymckormick
@jamespark3922 @CMeteor @XuGW-Kevin
[Benchmark] Add VL-RewardBench (#703)
276d90a 
* update vlrewardbench

* pre-commit fix

* formatter

* [Improvement] Better `AUTO_SPLIT` and model split for InternVL2

* [Minor] Improve CC-OCR Import

* [Model] Support QVQ

* [Model] Update Molmo Eval to Match Official Implementation (#648)

* add molmo prompts

* fix lint format

* [Fix] Refine Qwen-VL2 device assignment

* [Fix] Fix RealWorldQA md5

* update MMMU_DEV_VAL tsv

* [Fix] Fix confusing image width&height (#704)

Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>

* Update llama_vision.py (#705)

* [Fix] Fix Lint

* Fix Lint

* Fix Lint

---------

Co-authored-by: kennymckormick <dhd.efz@gmail.com>
Co-authored-by: jamespark3922 <jspark96@cs.washington.edu>
Co-authored-by: CMeteor <CMeteor@users.noreply.github.com>
Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>
Co-authored-by: Guowei Xu <113534787+XuGW-Kevin@users.noreply.github.com>
Mercury7353 pushed a commit to Mercury7353/VLMEvalKit that referenced this pull request Apr 28, 2025
@jamespark3922
[Model] Update Molmo Eval to Match Official Implementation (open-comp…
07d1a29 
…ass#648)

* add molmo prompts

* fix lint format
Mercury7353 pushed a commit to Mercury7353/VLMEvalKit that referenced this pull request Apr 28, 2025
@TobiasLee @kennymckormick
@jamespark3922 @CMeteor @XuGW-Kevin
[Benchmark] Add VL-RewardBench (open-compass#703)
1f5a104 
* update vlrewardbench

* pre-commit fix

* formatter

* [Improvement] Better `AUTO_SPLIT` and model split for InternVL2

* [Minor] Improve CC-OCR Import

* [Model] Support QVQ

* [Model] Update Molmo Eval to Match Official Implementation (open-compass#648)

* add molmo prompts

* fix lint format

* [Fix] Refine Qwen-VL2 device assignment

* [Fix] Fix RealWorldQA md5

* update MMMU_DEV_VAL tsv

* [Fix] Fix confusing image width&height (open-compass#704)

Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>

* Update llama_vision.py (open-compass#705)

* [Fix] Fix Lint

* Fix Lint

* Fix Lint

---------

Co-authored-by: kennymckormick <dhd.efz@gmail.com>
Co-authored-by: jamespark3922 <jspark96@cs.washington.edu>
Co-authored-by: CMeteor <CMeteor@users.noreply.github.com>
Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>
Co-authored-by: Guowei Xu <113534787+XuGW-Kevin@users.noreply.github.com>
Koii2k3 pushed a commit to wjnwjn59/VLMEvalKit that referenced this pull request Nov 13, 2025
@jamespark3922
[Model] Update Molmo Eval to Match Official Implementation (open-comp…
69589af 
…ass#648)

* add molmo prompts

* fix lint format
Koii2k3 pushed a commit to wjnwjn59/VLMEvalKit that referenced this pull request Nov 13, 2025
@TobiasLee @kennymckormick
@jamespark3922 @CMeteor @XuGW-Kevin
[Benchmark] Add VL-RewardBench (open-compass#703)
33941e3 
* update vlrewardbench

* pre-commit fix

* formatter

* [Improvement] Better `AUTO_SPLIT` and model split for InternVL2

* [Minor] Improve CC-OCR Import

* [Model] Support QVQ

* [Model] Update Molmo Eval to Match Official Implementation (open-compass#648)

* add molmo prompts

* fix lint format

* [Fix] Refine Qwen-VL2 device assignment

* [Fix] Fix RealWorldQA md5

* update MMMU_DEV_VAL tsv

* [Fix] Fix confusing image width&height (open-compass#704)

Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>

* Update llama_vision.py (open-compass#705)

* [Fix] Fix Lint

* Fix Lint

* Fix Lint

---------

Co-authored-by: kennymckormick <dhd.efz@gmail.com>
Co-authored-by: jamespark3922 <jspark96@cs.washington.edu>
Co-authored-by: CMeteor <CMeteor@users.noreply.github.com>
Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>
Co-authored-by: Guowei Xu <113534787+XuGW-Kevin@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kennymckormick kennymckormick kennymckormick approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jamespark3922 @kennymckormick