Add VidEoMT#44285
Merged
vasqu merged 77 commits intohuggingface:mainfrom Mar 25, 2026
Merged
Conversation
…-to-transformers [Videomt] Extend query-stage 5D/4D parity validation to 3-frame videos
…onversion [videomt] Improve verify adapters and DINOv3 failure diagnostics
…n-videomt [VidEoMT] Add temporal query-updater support, fix DINOv2 conversion mappings and re-verify yt_2019_vit_small
Contributor
|
What gpu @NielsRogge? Might be a diff between CI (A10) and your local one |
Contributor
|
run-slow: videomt |
Contributor
|
This comment contains models: ["models/videomt"] |
Contributor
|
I've added expectation for our CI devices so it works now, multi gpu still seems to fail. Would be nice if you could check @NielsRogge (general ci still shaky 😢) |
Collaborator
Author
|
@vasqu friendly ping |
Contributor
|
Please check #44285 (comment) It hasn't been addressed afaik |
Contributor
|
run-slow: videomt |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, videomt |
Contributor
|
This comment contains models: ["models/videomt"] |
vasqu
approved these changes
Mar 25, 2026
Contributor
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Fixed the last things, good to merge now 🤗
Contributor
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44285&sha=09b99a |
Contributor
|
Force merged, because the failure is known and I don't want to further delay this model |
zucchini-nlp
pushed a commit
to zucchini-nlp/transformers
that referenced
this pull request
Mar 27, 2026
* First draft * [Videomt] Extend query-stage parity checks to 3-frame inputs * [Videomt] Add full-model parity check against EoMT reference * [Videomt] Compare conversion against official GitHub reference * [Videomt] Simplify conversion to checkpoint-based HF mapping * [Videomt] Add --verify mode against upstream GitHub implementation * [Videomt] Improve --verify diagnostics with key remapping and layer checks * [Videomt] Improve verify backbone candidate fallback and remapping * [Videomt] Add DINOv3 verify compatibility patch and progress logging * [Videomt] Extend verify diagnostics with MLP/head parity checks * [Videomt] Make --verify succeed for converted weight mapping scope * [videomt] Improve verify adapters and candidate traceback diagnostics * [videomt] Adapt verify _pos_embed output for DINOv3 candidates * [videomt] Enable DINOv3 verify candidate by adapting EVA head_dim * [videomt] Add pre-query layer diagnostics to verify flow * [videomt] Add deterministic verify probes and deeper pre-query diffs * [videomt] Penalize skipped keys in verify candidate scoring * [videomt] Add no-rope A/B diagnostics to verify pre-query layers * [videomt] Add branch-level pre-query diagnostics to verify * [videomt] Add fine-grained MLP diagnostics to verify * [videomt] Verify layer-scale mapping parity in --verify * [videomt] Validate MLP diagnostic decomposition in verify * [videomt] Add token-group diagnostics for layer-4 MLP divergence * [VidEoMT] Add temporal query updater path and re-verify yt_2019_vit_small * [VidEoMT] Refine 5D execution order and re-check small checkpoint parity * Simplify conversion script and convert all dinov2 checkpoints * Add id2label mappings * Fix all tests * Add to auto mapping * Simplify verify_conversion_against_github_reference * Update absolute tolerance * Update date * Revert AGENTS.md * Address comments * Add circleci skill, fix circleci * Fix CI * Remove skills from git * Address comments * Address more comments * Address comment * Add docstrigns * Restore AGENTS.md * Address comment * fix this one * Address comments * [fix] mistral 4 docs (huggingface#44776) fix * Address comment * add expectations * Update date * Make fix-repo * fix multi gpu * fix with changes on main * fix date --------- Co-authored-by: vasqu <antonprogamer@gmail.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
NielsRogge
added a commit
to NielsRogge/transformers
that referenced
this pull request
Mar 30, 2026
* First draft * [Videomt] Extend query-stage parity checks to 3-frame inputs * [Videomt] Add full-model parity check against EoMT reference * [Videomt] Compare conversion against official GitHub reference * [Videomt] Simplify conversion to checkpoint-based HF mapping * [Videomt] Add --verify mode against upstream GitHub implementation * [Videomt] Improve --verify diagnostics with key remapping and layer checks * [Videomt] Improve verify backbone candidate fallback and remapping * [Videomt] Add DINOv3 verify compatibility patch and progress logging * [Videomt] Extend verify diagnostics with MLP/head parity checks * [Videomt] Make --verify succeed for converted weight mapping scope * [videomt] Improve verify adapters and candidate traceback diagnostics * [videomt] Adapt verify _pos_embed output for DINOv3 candidates * [videomt] Enable DINOv3 verify candidate by adapting EVA head_dim * [videomt] Add pre-query layer diagnostics to verify flow * [videomt] Add deterministic verify probes and deeper pre-query diffs * [videomt] Penalize skipped keys in verify candidate scoring * [videomt] Add no-rope A/B diagnostics to verify pre-query layers * [videomt] Add branch-level pre-query diagnostics to verify * [videomt] Add fine-grained MLP diagnostics to verify * [videomt] Verify layer-scale mapping parity in --verify * [videomt] Validate MLP diagnostic decomposition in verify * [videomt] Add token-group diagnostics for layer-4 MLP divergence * [VidEoMT] Add temporal query updater path and re-verify yt_2019_vit_small * [VidEoMT] Refine 5D execution order and re-check small checkpoint parity * Simplify conversion script and convert all dinov2 checkpoints * Add id2label mappings * Fix all tests * Add to auto mapping * Simplify verify_conversion_against_github_reference * Update absolute tolerance * Update date * Revert AGENTS.md * Address comments * Add circleci skill, fix circleci * Fix CI * Remove skills from git * Address comments * Address more comments * Address comment * Add docstrigns * Restore AGENTS.md * Address comment * fix this one * Address comments * [fix] mistral 4 docs (huggingface#44776) fix * Address comment * add expectations * Update date * Make fix-repo * fix multi gpu * fix with changes on main * fix date --------- Co-authored-by: vasqu <antonprogamer@gmail.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds the VidEoMT model, as described in VidEoMT: Your ViT is Secretly Also a Video Segmentation Model.
Gradio demo (running on ZeroGPU): https://huggingface.co/spaces/nielsr/videomt-transformers-demo
Original Github thread: tue-mps/videomt#1