[WIP] Add MM Grounding DINO by rziga · Pull Request #37925 · huggingface/transformers

rziga · 2025-05-02T12:55:27Z

What does this PR do?

It adds support for MM Grounding DINO and LLMDet (inference only).

I've added a weight conversion script and a modular transformers implementation.
The forward pass works with 1e-3 tolerance compared to the original (checked tiny and large checkpoints).
The tests are copied and modified from Grounding DINO, but two integration tests fail currently (batched consistency and CPU-GPU consistency).

TODO

[ ] fix failing tests

Added modular implementation for MM Grounding DINO from starting point created by add-new-model-like. Added conversion script from mmdetection to huggingface. TODO: Some tests are failing so that needs to be fixed.

…ion where box and class heads were not correctly assigned to inner model

Cross att masking and cpu-gpu consistency tests are still failing however.

yonigozlan · 2025-05-20T21:34:23Z

Hi @rziga ! Feel free to ping me when the PR is ready for review!

rziga · 2025-05-21T16:52:21Z

Sorry, I got busy with life stuff for the last couple of weeks, so I didn't have time to finish it up.

The model works, but I have to upload all the checkpoints to the hub and update the model doc before I mark it ready for review.

rziga · 2025-05-30T11:12:42Z

Pinging @yonigozlan and @qubvel as requested.

1himan · 2025-06-18T09:34:07Z

+-->
+
+# MM Grounding DINO
+


Model Documentations now are of a different format as per this issue right here. To standardize model cards and make it easy for the user.

1himan · 2025-06-27T08:30:51Z

Hey @rziga, I wanted to ask you that - how do you add a new model? Huggingface provides utilities to help you get started with the boilerplate, AFAIK this is where you get started - transformers add-new-model-like this CLI command will asks around 7-8 questions related to the model. The first one goes like this "What is the model you would like to duplicate? Please provide the lowercase model_type (e.g. roberta):"
this model will be used as a scaffolding to build the new model, we just have to change the main files which includes - architecture of the model, any pre/post processing utilities and a few more thing. We don't just dump in the research code(the original implementation of a model ) here. We first need to do a sort of code archaeology, remove the research specific code, and only add the code useful for inference.

Correct me if I'm wrong. And could you tell me your approach. 😄

rziga · 2025-06-27T09:10:46Z

Hi,

I don't really know if I'm the right person to ask, but yes, I started with transformers add-new-model-like. From then on I don't remember exactly, but it was a combination of following these two guides:

Since this modular approach is quite new, I also looked at other models that used modular transformers, like this one:

Adding RTDETRv2 #34773

Hope this helps somewhat. But again I don't think I'm the right person to ask here. It is probably better to open an issue to add a model and ask the actual contributors.

qubvel · 2025-06-27T12:58:03Z

Hey @rziga, I'm really sorry for the long delay. I'll review it next week. The modular code looks very minimal, so I expect we can merge it quickly! Thanks a lot for working on the model.

qubvel · 2025-07-30T11:16:42Z

Hi @rziga, I created org for MM Grounding DINO checkpoints and invited you

https://huggingface.co/openmmlab-community

qubvel · 2025-07-30T11:20:12Z

-        """
-        Preprocess an image or batch of images.
-        """


hmm, that's a bit strange, we should probably merge main and make repo-consistency to avoid it

yes, let's remove this unrelated change

ArthurZucker

Very nice! Thanks for making it so easy to review with modular!

ArthurZucker · 2025-07-30T11:24:10Z

+        if config.decoder_cls_embed_share:
+            _class_embed = MMGroundingDinoContrastiveEmbedding(config)
+            self.class_embed = nn.ModuleList([_class_embed for _ in range(config.decoder_layers)])
+        else:
+            module_list = []
+            for _ in range(config.decoder_layers):
+                _class_embed = MMGroundingDinoContrastiveEmbedding(config)
+                module_list.append(_class_embed)
+            self.class_embed = nn.ModuleList(module_list)
+
+        if config.decoder_bbox_embed_share:
+            _bbox_embed = MMGroundingDinoMLPPredictionHead(
+                input_dim=config.d_model, hidden_dim=config.d_model, output_dim=4, num_layers=3
+            )
+            self.bbox_embed = nn.ModuleList([_bbox_embed for _ in range(config.decoder_layers)])
+        else:
+            module_list = []
+            for _ in range(config.decoder_layers):
+                _bbox_embed = MMGroundingDinoMLPPredictionHead(
+                    input_dim=config.d_model, hidden_dim=config.d_model, output_dim=4, num_layers=3
+                )
+                module_list.append(_bbox_embed)
+            self.bbox_embed = nn.ModuleList(module_list)


in general we should not have so many code paths! Each case would lead to a different Embedding.... Are there a model for each case?

Both of these are always false, yes. I remember debating what to do. Cleanest thing would be to delete both decoder_bbox_embed_share and decoder_cls_embed_share and have no branching here. The issue is that original GroundingDino requires decoder_bbox_embed_share parameter, so I was kinda stuck with using it here as well.

@rziga if we are overriding the entire method, I suppose we can remove if/else code path here and remove these args from config (del decoder_bbox_embed_share in modular)

I tried to do that just now. The issue is that decoder_bbox_embed_share is used in a check inside config, so it doesn't get removed during modular conversion. Do I just copy the whole config class to remove the check?

ahh, I see! Yes, let's copy and modify the whole config class then

Do I keep the modified config in modular file or just move it to configuration file?

Never mind, I need to keep it in modular otherwise the conversion deletes it.

ArthurZucker · 2025-07-30T11:25:58Z

-        """
-        Preprocess an image or batch of images.
-        """


yes, let's remove this unrelated change

qubvel · 2025-07-31T08:58:27Z

+        self.backbone = MMGroundingDinoConvModel(backbone, position_embeddings)
+
+        # Create input projection layers
+        if config.num_feature_levels > 1:


Same here, can we remove this if/else? I suppose num_feature_levels > 1 for all checkpoints

Good catch. I'm not really sure why this if/else even is here. The first branch seems to produce the same thing as the second one.

qubvel · 2025-07-31T13:23:53Z

run-slow: mm_grounding_dino

github-actions · 2025-07-31T13:25:25Z

This comment contains run-slow, running the specified jobs:

models: ['models/mm_grounding_dino']
quantizations: [] ...

github-actions · 2025-08-01T13:53:01Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, grounding_dino, mm_grounding_dino

qubvel · 2025-08-01T13:55:28Z

run-slow: grounding_dino, mm_grounding_dino

github-actions · 2025-08-01T13:56:51Z

This comment contains run-slow, running the specified jobs:

models: ['models/grounding_dino', 'models/mm_grounding_dino']
quantizations: [] ...

qubvel · 2025-08-01T17:59:22Z

Huge thanks for the contribution @rziga! Great work 🤗

sushmanthreddy · 2025-08-07T16:38:24Z

@qubvel here support for llmdet is only added for the inference ,we have any support for training also ??can I pls take up the issue if its planned or open for community

* first commit Added modular implementation for MM Grounding DINO from starting point created by add-new-model-like. Added conversion script from mmdetection to huggingface. TODO: Some tests are failing so that needs to be fixed. * fixed a bug with modular definition of MMGroundingDinoForObjectDetection where box and class heads were not correctly assigned to inner model * cleaned up a hack in the conversion script * Fixed the expected values in integration tests Cross att masking and cpu-gpu consistency tests are still failing however. * changes for make style and quality * add documentation * clean up contrastive embedding * add mm grounding dino to loss mapping * add model link to config docstring * hack fix for mm grounding dino consistency tests * add special cases for unused config attr check * add all models and update docs * update model doc to the new style * Use super_kwargs for modular config * Move init to the _init_weights function * Add copied from for tests * fixup * update typehints * Fix-copies for tests * fix-copies * Fix init test * fix snippets in docs * fix consistency * fix consistency * update conversion script * fix nits in readme and remove old comments from conversion script * add license * remove unused config args * remove unnecessary if/else in model init * fix quality * Update references * fix test * fixup --------- Co-authored-by: qubvel <qubvel@gmail.com>

rziga added 7 commits May 2, 2025 13:56

first commit

7a8eb8f

Added modular implementation for MM Grounding DINO from starting point created by add-new-model-like. Added conversion script from mmdetection to huggingface. TODO: Some tests are failing so that needs to be fixed.

fixed a bug with modular definition of MMGroundingDinoForObjectDetect…

8aa1d0e

…ion where box and class heads were not correctly assigned to inner model

cleaned up a hack in the conversion script

b5750aa

Fixed the expected values in integration tests

6d1964c

Cross att masking and cpu-gpu consistency tests are still failing however.

changes for make style and quality

e1f8ec9

add documentation

ff56d8b

clean up contrastive embedding

036f6ec

qubvel added New model Vision labels May 5, 2025

rziga added 4 commits May 8, 2025 11:22

add mm grounding dino to loss mapping

043921e

add model link to config docstring

79bde49

hack fix for mm grounding dino consistency tests

515c1d6

add special cases for unused config attr check

d0daa27

add all models and update docs

36f003d

rziga marked this pull request as ready for review May 30, 2025 11:09

github-actions Bot requested review from ArthurZucker and Rocketknight1 May 30, 2025 11:10

1himan reviewed Jun 18, 2025

View reviewed changes

stevhliu mentioned this pull request Jun 18, 2025

[Community contributions] Model cards #36979

Closed

update model doc to the new style

caec85e

qubvel requested review from qubvel and removed request for ArthurZucker and Rocketknight1 July 16, 2025 09:38

Use super_kwargs for modular config

9e714c7

qubvel reviewed Jul 30, 2025

View reviewed changes

ArthurZucker approved these changes Jul 30, 2025

View reviewed changes

rziga added 2 commits July 30, 2025 14:29

add license

0c675fe

remove unused config args

e726f1d

qubvel reviewed Jul 31, 2025

View reviewed changes

rziga and others added 2 commits July 31, 2025 11:36

remove unnecessary if/else in model init

172cb17

Merge branch 'main' into add_mm_grounding_dino

32746d1

fix quality

68c64c9

qubvel self-requested a review August 1, 2025 10:04

qubvel added 3 commits August 1, 2025 13:17

Update references

e7ddf59

Merge branch 'main' into pr/rziga/37925

0a4c33a

fix test

78ea273

fixup

ce04714

qubvel approved these changes Aug 1, 2025

View reviewed changes

qubvel merged commit 3951d4a into huggingface:main Aug 1, 2025
25 of 26 checks passed

Conversation

rziga commented May 2, 2025

What does this PR do?

TODO

Uh oh!

yonigozlan commented May 20, 2025

Uh oh!

rziga commented May 21, 2025

Uh oh!

rziga commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1himan commented Jun 27, 2025

Uh oh!

rziga commented Jun 27, 2025

Uh oh!

qubvel commented Jun 27, 2025

Uh oh!

qubvel commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qubvel commented Jul 31, 2025

Uh oh!

github-actions Bot commented Jul 31, 2025

Uh oh!

github-actions Bot commented Aug 1, 2025

Uh oh!

qubvel commented Aug 1, 2025

Uh oh!

github-actions Bot commented Aug 1, 2025

Uh oh!

Uh oh!

qubvel commented Aug 1, 2025

Uh oh!

sushmanthreddy commented Aug 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants