Add SAM3-LiteText by NielsRogge · Pull Request #44320 · huggingface/transformers

NielsRogge · 2026-02-27T08:29:00Z

What does this PR do?

This PR adds SAM3-LiteText: An Anatomical Study of the SAM3 Text Encoder for Efficient Vision-Language Segmentation.

…to-transformers Integrate MobileCLIP-student LiteText encoder and add conversion tooling for EfficientSAM3 LiteText

…-to-transformers-fuvllg

…to-transformers-fuvllg Integrate SAM3-LiteText MobileCLIP student text encoder, conversion tooling, and parity/test fixes

HuggingFaceDocBuilderDev · 2026-02-27T08:38:16Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

NielsRogge · 2026-03-02T10:07:55Z


        # Get tokenizer class
        if self.lowercase_name in TOKENIZER_MAPPING_NAMES:
+            self.tokenizer_class = None


Note: have opened a separate PR for the CLI fixes here: #44334

JavierYepez · 2026-03-27T11:57:27Z

Maybe not the best practice to post this here but I'm getting an error when running
python utils/modular_model_converter.py sam3_lite_text

Converting src/transformers/models/sam3_lite_text/modular_sam3_lite_text.py to a single model single file format
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/javieryepez/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 1952, in run_converter
    converted_files = convert_modular_file(modular_file, source_library=source_library)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 1908, in convert_modular_file
    for file, module in create_modules(
                        ^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 1799, in create_modules
    nodes_to_add, file_type, new_imports = get_class_node_and_dependencies(modular_mapper, class_name, node, files)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 1733, in get_class_node_and_dependencies
    updated_node = replace_class_node(mapper, node, renamed_super_class, super_class)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 1084, in replace_class_node
    new_replacement_class = temp_module.visit(
                            ^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/module.py", line 89, in visit
    result = super(Module, self).visit(visitor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/module.py", line 74, in _visit_and_replace_children
    body=visit_body_sequence(self, "body", self.body, visitor),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 227, in visit_body_sequence
    return tuple(visit_body_iterable(parent, fieldname, children, visitor))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 193, in visit_body_iterable
    new_child = child.visit(visitor)
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/statement.py", line 1989, in _visit_and_replace_children
    body=visit_required(self, "body", self.body, visitor),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 81, in visit_required
    result = node.visit(visitor)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/statement.py", line 704, in _visit_and_replace_children
    body=visit_body_sequence(self, "body", self.body, visitor),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 227, in visit_body_sequence
    return tuple(visit_body_iterable(parent, fieldname, children, visitor))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 193, in visit_body_iterable
    new_child = child.visit(visitor)
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 237, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/.venv/lib/python3.11/site-packages/libcst/_visitors.py", line 71, in on_leave
    updated_node = leave_func(original_node, updated_node)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 365, in leave_FunctionDef
    original_modeling_method_body = self.original_modeling_methods[func_name].body.body
                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: '__init__'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/javieryepez/Documents/Projects/opensource/transformers/utils/modular_model_converter.py", line 2039, in <module>
    pool.map(partial(run_converter, source_library=args.source_library), dependency_level_files)
  File "/Users/javieryepez/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/javieryepez/.local/share/uv/python/cpython-3.11.14-macos-aarch64-none/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
KeyError: '__init__'

vasqu · 2026-04-02T17:50:35Z

ping me when it's ready for review 🤗 not sure atm :p

yonigozlan · 2026-04-06T20:29:06Z

Thanks for reviewing @vasqu ! It should be ready to merge now ;)

vasqu

Sorry, found a few other smaller things. Shouldn't be anything big (and also some repeating stuff)

vasqu · 2026-04-10T07:43:42Z

+model = AutoModel.from_pretrained("yonigozlan/sam3-litetext-s0", device_map="auto")
+processor = AutoProcessor.from_pretrained("yonigozlan/sam3-litetext-s0")


Are there any plans to move these to another repo?

vasqu

I'm running slow tests in a second, I really only have the last nits left

Should be mergable today or tomorrow if we are fast enough :)

vasqu · 2026-04-13T11:26:31Z

+    window_size (`int`, *optional*, defaults to 24):
+        Window size for windowed attention.
+    global_attn_indexes (`list[int]`, *optional*, defaults to `[7, 15, 23, 31]`):
+        Indexes of layers with global attention.


These are for the underlying auto models ig?

vasqu · 2026-04-13T11:29:02Z

+    rope_theta (`float`, *optional*, defaults to 10000.0):
+        Base frequency for RoPE.


Should not be used at all, if anything we should change default theta (which is 10_000.0 already so no need to change)

It is generated by make fix-repo

I see that it is indeed needed either way for the sam vit model but we should probably refactor this a bit to follow more standard RoPE implementations cc @yonigozlan

vasqu · 2026-04-13T11:43:26Z

run-slow: sam3_lite_text

github-actions · 2026-04-13T11:44:45Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/sam3_lite_text"]
quantizations: []

github-actions · 2026-04-13T11:49:26Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	8a21b5fc	workflow commit (merge commit)
PR	fe153ad8	branch commit (from PR)
main	a001f344	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu · 2026-04-13T17:40:49Z

run-slow: sam3, sam3_lite_text

github-actions · 2026-04-13T17:41:46Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, sam3, sam3_lite_text, sam3_video

github-actions · 2026-04-13T17:42:21Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/sam3", "models/sam3_lite_text"]
quantizations: []

github-actions · 2026-04-13T17:50:58Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	38f4d2a3	workflow commit (merge commit)
PR	ccb49029	branch commit (from PR)
main	52f2268b	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

yonigozlan · 2026-04-13T18:37:07Z

Nice! I'll reach out to the authors to transfer the checkpoints

* Fix * First draft * Add push-to-hub options for SAM3-LiteText conversion * Fix SAM3-LiteText model tests and text encoder init stability * Add LiteText ViT auto mappings and use LiteText config * Improve conversion script * Do not require triton * Improve modeling * Fix repo * Fix repo * Add vision model to auto mapping * Add missing entries to auto mapping * reverse serve.py * simplify implementation * fix modular * Address review comments * fix repo * fix after review 2 * fix tests + repo * Address comments * Address comments * Make fix-repo * add to hub cache + fixup base sam3 as well --------- Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co> Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com> Co-authored-by: vasqu <antonprogamer@gmail.com>

NielsRogge and others added 10 commits February 26, 2026 18:10

Fix

4bae67b

First draft

2c96661

Add push-to-hub options for SAM3-LiteText conversion

490ff1f

Merge pull request #69 from NielsRogge/codex/add-sam3-litetext-model-…

00777a5

…to-transformers Integrate MobileCLIP-student LiteText encoder and add conversion tooling for EfficientSAM3 LiteText

Fix SAM3-LiteText model tests and text encoder init stability

4dd3735

Add LiteText ViT auto mappings and use LiteText config

0d96394

Merge branch 'add_sam_3_lite_text' into codex/add-sam3-litetext-model…

06fbf45

…-to-transformers-fuvllg

Merge pull request #70 from NielsRogge/codex/add-sam3-litetext-model-…

53f7dd4

…to-transformers-fuvllg Integrate SAM3-LiteText MobileCLIP student text encoder, conversion tooling, and parity/test fixes

Improve conversion script

4d8008a

Do not require triton

a5ce4ca

NielsRogge and others added 8 commits February 27, 2026 10:18

Improve modeling

db44153

Fix repo

98cea30

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

5fdc242

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

dcaceff

Fix repo

c85571e

Merge branch 'main' into add_sam_3_lite_text

8ba6455

Add vision model to auto mapping

5ab59a6

Add missing entries to auto mapping

d5728ae

NielsRogge commented Mar 2, 2026

View reviewed changes

yonigozlan added the New model label Mar 5, 2026

yonigozlan added 2 commits March 24, 2026 18:49

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

813dd0b

reverse serve.py

583df21

JavierYepez mentioned this pull request Mar 30, 2026

[SAM3-LiteText] Fix modular converter KeyError and torchvision soft dependency NielsRogge/transformers#72

Closed

NielsRogge force-pushed the add_sam_3_lite_text branch from 4156a0f to e5a5063 Compare March 30, 2026 16:28

simplify implementation

8f35675

yonigozlan force-pushed the add_sam_3_lite_text branch from e5a5063 to 8f35675 Compare March 30, 2026 16:59

yonigozlan added 2 commits March 30, 2026 16:59

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

37c3fcd

fix modular

a402d1a

yonigozlan added 3 commits April 6, 2026 20:02

fix after review 2

ac3370a

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

7204251

fix tests + repo

9baf14d

yonigozlan requested a review from vasqu April 6, 2026 20:29

Merge branch 'main' into add_sam_3_lite_text

5aaad48

vasqu approved these changes Apr 10, 2026

View reviewed changes

NielsRogge added 2 commits April 11, 2026 18:17

Merge remote-tracking branch 'upstream/main' into add_sam_3_lite_text

c5deac2

Address comments

fe153ad

vasqu approved these changes Apr 13, 2026

View reviewed changes

NielsRogge and others added 4 commits April 13, 2026 15:46

Address comments

a1a9c1e

Make fix-repo

6ffbc8e

Merge branch 'main' into add_sam_3_lite_text

9570b50

add to hub cache + fixup base sam3 as well

ccb4902

vasqu added this pull request to the merge queue Apr 13, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 13, 2026

vasqu added this pull request to the merge queue Apr 13, 2026

Merged via the queue into huggingface:main with commit 22278df Apr 13, 2026
29 checks passed

yonigozlan mentioned this pull request Apr 20, 2026

[Sam3LiteText] Remove unnecessary modules/configs #45535

Merged

		model = AutoModel.from_pretrained("yonigozlan/sam3-litetext-s0", device_map="auto")
		processor = AutoProcessor.from_pretrained("yonigozlan/sam3-litetext-s0")

		rope_theta (`float`, optional, defaults to 10000.0):
		Base frequency for RoPE.

Conversation

NielsRogge commented Feb 27, 2026

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Feb 27, 2026

Uh oh!

NielsRogge Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

JavierYepez commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasqu commented Apr 2, 2026

Uh oh!

yonigozlan commented Apr 6, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vasqu Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

NielsRogge Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

CI Results

Commit Info

Uh oh!

vasqu commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

CI Results

Commit Info

Uh oh!

Uh oh!

Uh oh!

yonigozlan commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

JavierYepez commented Mar 27, 2026 •

edited

Loading