Adding RTDETRv2 by jadechoghari · Pull Request #34773 · huggingface/transformers

jadechoghari · 2024-11-18T10:04:16Z

What does this PR do?

This PR add RTDETRv2 into the Transformers library. There is a new thing in transformers called modular, which adds new models by creating a modeling_modelname.py file. Since RTDETRv2 only updates the decoder part while keeping the rest of the model unchanged, it serves as an ideal use case for this modular approach.

What’s Left:

Fix the modular -> modeling cookie cutter setup
Remove the scratch folder (auto-generated by the add-model cookie cutter)
Add support for resnet

Colab to replicate original author logits: https://colab.research.google.com/drive/1Vql-9JuFKz7N7l83NmHPP2E1ZyGZnpzX?usp=sharing

SangbumChoi · 2024-11-18T10:11:45Z

@jadechoghari Hi thanks for working this, original PR for RTDetr-v2 can be found here. Perhaps you can fill the blank based on this
#33244

jadechoghari · 2024-11-18T10:27:22Z

Hi @SangbumChoi, yes - this is a new PR since we're using a new concept called modular.
also, thank you so much for your work - lot of stuff here is based on it especially the v1 implementation!

SangbumChoi · 2024-11-18T10:31:42Z

Great. FYI I also have confirmed that there is slight tolerance issue of converting v2 weight since it has difference in MSDA.

jadechoghari · 2024-11-18T10:40:00Z

yes nw, I checked your PR!

Cyrilvallez

Hey! Here is a first round or reviews! 🤗 I hope it clears some confusions about modular!
Also, let's be consistent with the library: let's rename the files and folders with rt_detr_v2 instead of rtdetrv2, and use CamelCase for class names, i.e. RtDetrV2Class instead of RTDetrv2Class.

jadechoghari · 2024-11-21T18:01:24Z

great! - i'll update accordingly - thx!

Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>

jadechoghari · 2024-11-21T18:44:58Z

Hey @Cyrilvallez, lmk if anything else needs to be updated for Modular - i've incorporated all your suggestions and fixes!

Cyrilvallez

Ok! As discussed in Slack and this review, let's first take care of all the naming conventions in the folder/file/class names 🤗
Also, add/modify all the needed remaining files!
Then, please run

make fixup
python modular_model_converter.py --files_to_parse src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py

from the root of the transformers repo to take care of the coding style and convert your modular!
Once this is done, we'll discuss potential improvements for the modular itself!

jadechoghari · 2024-11-27T06:17:14Z

Changes updated @Cyrilvallez !
Adding the make fixup error:

utils/check_modular_conversion.py
Traceback (most recent call last):
  File "/home/user/app/transformers/utils/check_modular_conversion.py", line 75, in <module>
    non_matching_files += compare_files(modular_file_path, args.fix_and_overwrite)
  File "/home/user/app/transformers/utils/check_modular_conversion.py", line 55, in compare_files
    generated_modeling_content = convert_modular_file(modular_file_path)
  File "/home/user/app/transformers/utils/modular_model_converter.py", line 1447, in convert_modular_file
    for file, module in create_modules(cst_transformers).items():
  File "/home/user/app/transformers/utils/modular_model_converter.py", line 1387, in create_modules
    nodes_to_add, file_type, new_imports = get_class_node_and_dependencies(modular_mapper, class_name, node, files)
  File "/home/user/app/transformers/utils/modular_model_converter.py", line 1348, in get_class_node_and_dependencies
    relative_dependency_order = mapper.compute_relative_order(all_dependencies_to_add)
  File "/home/user/app/transformers/utils/modular_model_converter.py", line 739, in compute_relative_order
    remaining_dependencies.remove(class_name)
KeyError: 'RtDetrV2MultiheadAttention'
make: *** [Makefile:39: repo-consistency] Error 1

Cyrilvallez

Hey! Indeed because the order of classes in modeling_rt_detr.py is messy (some class are used before being defined), the modular was failing! You can pull from #35056 for the fix

See comments, we're getting there! 🤗

EDIT: also, please run make fixup to clean-up the files a bit 🙏

qubvel · 2025-02-02T08:59:08Z

run-slow: rt_detr_v2

github-actions · 2025-02-02T09:00:28Z

This comment contains run-slow, running the specified jobs: ['models/rt_detr_v2'] ...

qubvel

Thanks for working on the model! A few comments on my side:

… into rtdetrv2

ArthurZucker

feel free to merge when you are happy @qubvel

qubvel · 2025-02-05T12:00:47Z

Hey @jadechoghari, I see the logits in tests are changed significantly after removing custom kernel attention, please, ensure it matches with the original implementation.

jadechoghari · 2025-02-05T15:20:53Z

@qubvel ofc, i've added a notebook at the top of the PR to replicate the original author logits which matches the one in the testing :)!

qubvel · 2025-02-06T18:13:28Z

run-slow: rt_detr, rt_detr_v2

github-actions · 2025-02-06T18:14:48Z

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...

HuggingFaceDocBuilderDev · 2025-02-06T18:40:09Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

qubvel · 2025-02-06T19:35:04Z

@jadechoghari congratulations on the second model merged 🤗 great work 🚀

* cookiecutter add rtdetrv2 * make modular working * working modelgit add . * working modelgit add . * finalize moduar inheritence * finalize moduar inheritence * Update src/transformers/models/rtdetrv2/modular_rtdetrv2.py Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> * update modular and add rename * remove output ckpt * define loss_kwargs * fix CamelCase naming * fix naming + files * fix modular and convert file * additional changes * fix modular * fix import error (switch to lazy) * fix autobackbone * make style * add * update testing * fix loss * remove old folder * fix testing for v2 * update docstring * fix docstring * add resnetv2 (with modular bug to fix) * remove resnetv2 backbone * fix changes * small fixes * remove rtdetrv2resnetconfig * add rtdetrv2 name to convert * make style * Update docs/source/en/model_doc/rt_detr_v2.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * fix modular typo after review * add reviewed changes * add final review changes * Update docs/source/en/model_doc/rt_detr_v2.md Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> * Update src/transformers/models/rt_detr_v2/__init__.py Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> * Update src/transformers/models/rt_detr_v2/convert_rt_detr_v2_weights_to_hf.py Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> * add review changes * remove rtdetrv2 resnet * removing this weird project change * change ckpt name from jadechoghari to author * implement review and update testing * update naming and remove wrong ckpt * name * make fix-copies * Fix RT-DETR loss * Add resources, fix name * Fix repo in docs * Fix table name --------- Co-authored-by: jadechoghari <jadechoghari@users.noreply.huggingface.co> Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: qubvel <qubvel@gmail.com>

jadechoghari added 6 commits November 15, 2024 17:10

cookiecutter add rtdetrv2

457ec3c

make modular working

53dd35b

working modelgit add .

0dcc0c5

working modelgit add .

13ed8e5

finalize moduar inheritence

f1cf051

finalize moduar inheritence

40e4c20

qubvel added New model Vision run-slow Modular labels Nov 18, 2024

Cyrilvallez reviewed Nov 20, 2024

View reviewed changes

jadechoghari and others added 4 commits November 21, 2024 13:01

Update src/transformers/models/rtdetrv2/modular_rtdetrv2.py

52e9d8f

Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>

update modular and add rename

1cb8414

remove output ckpt

dae9513

define loss_kwargs

e940d27

Cyrilvallez self-requested a review November 22, 2024 10:28

fix CamelCase naming

45349a6

Cyrilvallez reviewed Nov 25, 2024

View reviewed changes

fix naming + files

cd6ffca

Cyrilvallez reviewed Dec 3, 2024

View reviewed changes

SangbumChoi mentioned this pull request Dec 16, 2024

Request to add D-FINE #35283

Closed

2 tasks

jadechoghari added 2 commits December 27, 2024 18:35

fix modular and convert file

d762677

additional changes

03bb2cc

Merge branch 'main' into rtdetrv2

6bfdc76

qubvel self-requested a review February 3, 2025 11:31

qubvel reviewed Feb 3, 2025

View reviewed changes

Comment thread docs/source/en/model_doc/rt_detr_v2.md Outdated

Comment thread src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py Outdated

Comment thread src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py Outdated

Comment thread tests/models/rt_detr_v2/test_modeling_rt_detr_v2.py

jadechoghari added 2 commits February 4, 2025 03:24

implement review and update testing

fed0d54

Merge branch 'rtdetrv2' of https://github.com/jadechoghari/transformers…

8d93109

… into rtdetrv2

ArthurZucker approved these changes Feb 4, 2025

View reviewed changes

jadechoghari added 3 commits February 5, 2025 22:53

update naming and remove wrong ckpt

cb7bf37

name

83711d4

make fix-copies

d306d92

qubvel self-requested a review February 6, 2025 13:03

jadechoghari and others added 2 commits February 6, 2025 07:06

Merge branch 'main' into rtdetrv2

f344181

Fix RT-DETR loss

ade4e3f

qubvel added 3 commits February 6, 2025 18:48

Add resources, fix name

dd3c155

Fix repo in docs

aedcf12

Fix table name

13b1c47

qubvel merged commit 006d924 into huggingface:main Feb 6, 2025

jadechoghari deleted the rtdetrv2 branch February 6, 2025 19:32

daniel-bogdoll mentioned this pull request Feb 12, 2025

RT-DETR v2 mcity/mcity_data_engine#60

Open

rziga mentioned this pull request Jun 27, 2025

[WIP] Add MM Grounding DINO #37925

Merged

Conversation

jadechoghari commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

What’s Left:

Uh oh!

SangbumChoi commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jadechoghari commented Nov 18, 2024

Uh oh!

SangbumChoi commented Nov 18, 2024

Uh oh!

jadechoghari commented Nov 18, 2024

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jadechoghari commented Nov 21, 2024

Uh oh!

jadechoghari commented Nov 21, 2024

Uh oh!

Cyrilvallez left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jadechoghari commented Nov 27, 2024

Uh oh!

Cyrilvallez left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qubvel commented Feb 2, 2025

Uh oh!

github-actions Bot commented Feb 2, 2025

Uh oh!

qubvel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

qubvel commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jadechoghari commented Feb 5, 2025

jadechoghari commented Nov 18, 2024 •

edited

Loading

SangbumChoi commented Nov 18, 2024 •

edited

Loading

Cyrilvallez left a comment •

edited

Loading

Cyrilvallez left a comment •

edited

Loading

qubvel commented Feb 5, 2025 •

edited

Loading