Skip to content

Add full GGUF loading support for GPT‑OSS (fixes #43366, supersedes #43757) latest#45500

Closed
sirzechs66 wants to merge 6 commits intohuggingface:mainfrom
sirzechs66:gpt-oss-support
Closed

Add full GGUF loading support for GPT‑OSS (fixes #43366, supersedes #43757) latest#45500
sirzechs66 wants to merge 6 commits intohuggingface:mainfrom
sirzechs66:gpt-oss-support

Conversation

@sirzechs66
Copy link
Copy Markdown
Contributor

What does this PR do?

This PR adds full GGUF loading support for GPT‑OSS models (20B/120B). It allows Transformers (and consequently vLLM) to directly load GPT‑OSS GGUF files without falling back to a wrong architecture. The changes include:

  • Architecture registration in GGUF mappings.
  • A custom GptOssTensorProcessor to handle MoE expert splitting and gate/up interleaving.
  • Reconstruction of nested rope_scaling (YaRN) from flat GGUF metadata.
  • Tests: fast registration test + slow integration test using a real 20B GGUF file.

Fixes #43366, supersedes #43757.
Related vLLM issue: vllm-project/vllm#22353

Code Agent Policy

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). – Not applicable, it adds a feature.
  • Did you read the contributor guideline, Pull Request section? – Yes.
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. – Issue GGUF model with architecture gpt-oss support #43366, discussion in comments.
  • Did you make sure to update the documentation with your changes? – yes!
  • Did you write any new necessary tests? – Yes, in test/quantization/test_ggml.py

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Tagging:

@github-actions
Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: ggml

@github-actions
Copy link
Copy Markdown
Contributor

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45500&sha=483e8f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GGUF model with architecture gpt-oss support

1 participant