feat: Add GGUF loading support for Llama 4 (text) by garybadwal · Pull Request #45546 · huggingface/transformers

garybadwal · 2026-04-21T08:58:51Z

What does this PR do?

Adds GGUF loading support for the Llama 4 text backbone (llama4_text), the last major Llama-family architecture that still raised GGUF model with architecture llama4 is not supported yet. Users can now do:

AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF",
    gguf_file="Llama-4-Scout-17B-16E-Instruct-Q2_K_L.gguf",
)

Scope is text-only, matches what llama.cpp's convert_hf_to_gguf.py emits for Llama 4 (vision tensors are skipped there), and is consistent with the existing gemma3_text GGUF path.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@SunMarc — you own #33260 and the GGUF loader; would appreciate your review.
cc @MekkCyber @Isotr0py — you weighed in on #36144 (DeepSeek GGUF) and have context on the GGUF MoE pipeline.
cc @ArthurZucker — as maintainer of the Llama 4 text model in case anything about the MoE / tokenizer path looks off.

…ssing

SunMarc

Left a comment

SunMarc · 2026-04-21T12:25:45Z

+
+    def test_llama4_config_mapping(self):
+        """Test that Llama 4 GGUF config mapping is correctly registered."""
+        from transformers.integrations.ggml import GGUF_CONFIG_MAPPING
+
+        self.assertIn("llama4", GGUF_CONFIG_MAPPING)
+        mapping = GGUF_CONFIG_MAPPING["llama4"]
+
+        expected_mappings = {
+            "context_length": "max_position_embeddings",
+            "block_count": "num_hidden_layers",
+            "feed_forward_length": "intermediate_size_mlp",
+            "expert_feed_forward_length": "intermediate_size",
+            "embedding_length": "hidden_size",
+            "rope.freq_base": "rope_theta",
+            "attention.key_length": "head_dim",
+            "attention.head_count": "num_attention_heads",
+            "attention.head_count_kv": "num_key_value_heads",
+            "attention.layer_norm_rms_epsilon": "rms_norm_eps",
+            "vocab_size": "vocab_size",
+            "expert_count": "num_local_experts",
+            "expert_used_count": "num_experts_per_tok",


those tests are a bit weak, not sure that passing the expected_mappings is a good enough test. we need to test the model and the tokenizer.

addressed 👍🏻

SunMarc · 2026-04-21T12:25:56Z

+    def test_llama4_architecture_mapping(self):
+        """Test that Llama 4 text-only GGUFs route to GGUFLlamaConverter and Llama4TensorProcessor."""
+        from transformers.integrations.ggml import GGUF_TO_FAST_CONVERTERS, GGUFLlamaConverter
+        from transformers.modeling_gguf_pytorch_utils import TENSOR_PROCESSORS, Llama4TensorProcessor
+
+        self.assertIn("llama4_text", GGUF_TO_FAST_CONVERTERS)
+        self.assertEqual(GGUF_TO_FAST_CONVERTERS["llama4_text"], GGUFLlamaConverter)
+        self.assertIn("llama4", TENSOR_PROCESSORS)
+        self.assertEqual(TENSOR_PROCESSORS["llama4"], Llama4TensorProcessor)
+


addressed 👍🏻

SunMarc · 2026-04-21T12:26:10Z

+        # Llama 4 is large and heavily quantised; we only check that the load path works end-to-end
+        # and produces a non-empty decoded string rather than asserting exact text.
+        decoded = tokenizer.decode(out[0], skip_special_tokens=True)
+        self.assertTrue(len(decoded) > len(self.example_text))


this is not enough for a test

addressed 👍🏻

garybadwal · 2026-04-21T14:38:26Z

Thanks @SunMarc, I'll check these and add more strong test case in this.

Again thank you for your precision time. 🫡

…move deprecated config mappings

github-actions · 2026-04-25T06:32:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: ggml

github-actions · 2026-04-25T06:47:11Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45546&sha=44761d

garybadwal · 2026-04-25T06:47:49Z

Hey @SunMarc, thanks for the feedback! I've addressed all three comments:

Removed test_llama4_config_mapping and test_llama4_architecture_mapping as suggested.
Added a proper test_llama4_q2_k_l_tokenizer test that saves/reloads the tokenizer and validates round-trip encoding.
Updated test_llama4_q2_k_l to assert against an exact EXPECTED_TEXT string instead of the weak length check.

Regarding the CI failure, it's in GlmImageModelTest::test_sample_generate_dict_output which is a pre-existing tensor size mismatch in modeling_glm_image.py.

My PR only touches ggml.py, modeling_gguf_pytorch_utils.py, and the GGUF test file, so this failure is unrelated to my changes. But if there is anything that I can do about this please let me know.

feat: add Llama 4 support with configuration mapping and tensor proce…

483eeeb

…ssing

SunMarc reviewed Apr 21, 2026

View reviewed changes

garybadwal and others added 2 commits April 25, 2026 11:59

refactor: update Llama 4 tests to improve tokenizer validation and re…

68cdef0

…move deprecated config mappings

Merge branch 'main' into feature/llama4-gguf-loading

44761d2

garybadwal requested a review from SunMarc April 25, 2026 06:48

evalstate mentioned this pull request Apr 28, 2026

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#41

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add GGUF loading support for Llama 4 (text)#45546

feat: Add GGUF loading support for Llama 4 (text)#45546
garybadwal wants to merge 3 commits intohuggingface:mainfrom
garybadwal:feature/llama4-gguf-loading

garybadwal commented Apr 21, 2026

Uh oh!

SunMarc left a comment

Uh oh!

SunMarc Apr 21, 2026

Uh oh!

garybadwal Apr 25, 2026

Uh oh!

SunMarc Apr 21, 2026

Uh oh!

garybadwal Apr 25, 2026

Uh oh!

SunMarc Apr 21, 2026

Uh oh!

garybadwal Apr 25, 2026

Uh oh!

garybadwal commented Apr 21, 2026

Uh oh!

github-actions Bot commented Apr 25, 2026

Uh oh!

github-actions Bot commented Apr 25, 2026

Uh oh!

garybadwal commented Apr 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

garybadwal commented Apr 21, 2026

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

SunMarc Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

garybadwal Apr 25, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

garybadwal Apr 25, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

garybadwal Apr 25, 2026

Choose a reason for hiding this comment

Uh oh!

garybadwal commented Apr 21, 2026

Uh oh!

github-actions Bot commented Apr 25, 2026

Uh oh!

github-actions Bot commented Apr 25, 2026

Uh oh!

garybadwal commented Apr 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants