Skip to content

Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models#2001

Merged
echarlaix merged 28 commits intomainfrom
add-hiera-onnx
Dec 19, 2024
Merged

Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models#2001
echarlaix merged 28 commits intomainfrom
add-hiera-onnx

Conversation

@xenova
Copy link
Copy Markdown
Contributor

@xenova xenova commented Aug 29, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@xenova xenova changed the title Add ONNX export support for Hiera models Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models Aug 30, 2024
@SangbumChoi
Copy link
Copy Markdown

@xenova Hi xenova! I think you missed to add official support for maskformer

ONNX Model for semantic-segmentation, with an all-MLP decode head on top e.g. for ADE20k, CityScapes. This class officially supports segformer.

Copy link
Copy Markdown
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delay @xenova !! Looks so good, thanks for adding so many new models ❤️

Comment thread optimum/exporters/tasks.py Outdated
Comment on lines +1207 to +1208
"feature-extraction",
"masked-im",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "image-feature-extraction" (or "feature-extraction") should be enough https://huggingface.co/datasets/huggingface/transformers-metadata/blob/main/pipeline_tags.json#L1128

Suggested change
"feature-extraction",
"masked-im",
"feature-extraction",

if not the correct class needed to load the model can be added to _CUSTOM_CLASSES : for vit-mae I only see AutoModel or AutoModelForPreTraining

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 7a2e94a

Comment thread optimum/exporters/tasks.py Outdated
"vit-msn": supported_tasks_mapping(
"feature-extraction",
"image-classification",
"masked-im",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same I think can be removed :

Suggested change
"masked-im",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 01929b2

@echarlaix
Copy link
Copy Markdown
Collaborator

echarlaix commented Dec 18, 2024

Also seems there is an issue for Maskformer for the feature-extraction task

FAILED exporters/onnx/test_exporters_onnx_cli.py::OnnxCLIExportTestCase::test_exporters_cli_pytorch_cpu_357_maskformer_feature_extraction_default_hf_internal_testing_tiny_random_MaskFormerForInstanceSegmentation - RuntimeError: number of output names provided (1) exceeded number of outputs (0)

the expected output name seems to be "transformer_decoder_last_hidden_state" https://github.com/huggingface/transformers/blob/69e31eb1bf123d5bd0183b753da83ab7dd9c2eec/src/transformers/models/maskformer/modeling_maskformer.py#L1615 while feature-extraction gives "last_hidden_state", can be fixed by updating MaskFormerOnnxConfig

edit : should be fixed with 3fa346c

@echarlaix echarlaix merged commit 0c42291 into main Dec 19, 2024
@echarlaix echarlaix deleted the add-hiera-onnx branch December 19, 2024 08:47
@xenova
Copy link
Copy Markdown
Contributor Author

xenova commented Dec 19, 2024

Thanks so much @echarlaix! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants