Add Solar-Open Model by oesni · Pull Request #43244 · huggingface/transformers

oesni · 2026-01-13T04:53:03Z

What does this PR do?

Implements Solar-Open model.
Solar Open is the open-weights MoE Solar LLM created by Upstage.

model repo: https://huggingface.co/upstage/Solar-Open-100B
technical report: https://huggingface.co/papers/2601.07022

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

vasqu · 2026-01-13T13:37:46Z

@oesni you can ping me when you think it's ready for review (assuming it's not yet because it's a draft)

oesni · 2026-01-14T08:53:13Z

It's ready for review! @vasqu
But after rebase, it seems some tests fails.
I'll work on it.

oesni · 2026-01-14T09:14:19Z

wonder if it's okay to add SolarOpenConfig class to OBJECTS_TO_IGNORE in utils/check_docstrings.py since auto-generated config class fails docstring check.
There is comment for OBJECTS_TO_IGNORE: Do not add anything here ...

vasqu

Looks already super good, my main points are mostly related to making the config more aligned with the current way we handle rope + tests to add a small dummy model for us - 100B is sadly too heavy for our CI 😢

vasqu · 2026-01-14T18:02:23Z

+rendered properly in your Markdown viewer.
+
+-->
+*This model was released on 2025-12-31 and added to Hugging Face Transformers on 2026-01-13.*


Just as reminder to keep track of this when we merge

It's now enforced on our CI, will need make fix-repo but that happens automatically then

vasqu · 2026-01-14T18:48:59Z

wonder if it's okay to add SolarOpenConfig class to OBJECTS_TO_IGNORE in utils/check_docstrings.py since auto-generated config class fails docstring check.
There is comment for OBJECTS_TO_IGNORE: Do not add anything here ...

Just checked why it failed, we should not add it there. You can run make fix-repo and you will see that it complains because the config has a wrong default for rope_parameters but since we will change it (we should default to None --> non-mutable args) either way let's wait. We can take a look afterwards or I just quickly fix that no worries

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

vasqu

Super small nits, let's move the test(s) under causal lm tester, mostly the one test to check if partial rotary factor has the correct default

vasqu · 2026-01-19T14:49:49Z

+        attention_bias (`bool`, *optional*, defaults to `False`):
+            Whether to use a bias in the projection layers.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.


We usually only support extra branches / features when they are actually used within a model

vasqu · 2026-01-19T15:08:48Z

Yup, dont worry about the CI - it's been a bit flaky these past few days/weeks

This reverts commit 9023688.

This reverts commit 3c275dc.

This reverts commit e6adcd9.

This reverts commit 573fa9a.

vasqu · 2026-01-21T15:48:56Z

run-slow: solar_open

github-actions · 2026-01-21T15:50:16Z

This comment contains run-slow, running the specified jobs:

models: ["models/solar_open"]
quantizations: []

github-actions · 2026-01-21T16:03:20Z

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

github-actions · 2026-01-21T16:18:57Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, solar_open

vasqu · 2026-01-21T16:23:41Z

run-slow: solar_open

github-actions · 2026-01-21T16:25:12Z

This comment contains run-slow, running the specified jobs:

models: ["models/solar_open"]
quantizations: []

github-actions · 2026-01-21T16:32:30Z

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

vasqu · 2026-01-21T16:40:22Z

Merging now 🤗 thanks for the contribution

HuggingFaceDocBuilderDev · 2026-01-21T16:45:47Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

oesni · 2026-01-22T02:20:41Z

Thanks for the review! 🤗 @vasqu

@vasqu

* feat: implement solar-open-100b * feat: update modeling_solar_open.py * feat: update solar-open config * chore: apply style * feat: remove _tied_weights_keys * feat: update modeling code * chore: remove speech_to_text_2 in modeling * docs: solar_open model * test: solar open model * chore: re-convert modular * fix: remove require_read_token * Apply suggestion from @vasqu Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * chore: update lincse year -> 2026 * feat: add solar_open to tokenizer mapping * chore: update license year * test: remove _torch_compile_train_cls * docs: update solar_open doc * refactor: simplify SolarOpenDecoderLayer * refactor: inherit Glm4MoeConfig class * fix: handle head_dim properly * chore: apply style * fix: default parameters * test: use tiny dummy model * update expectations and switch to eager moe (no fluctuations per grouped_mm / batched_mm) * chore: remove trust_remote_code (suggestion from @vasqu) Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/solar_open/modular_solar_open.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * chore: update config docstring * chore: add partial_rotary_factor workaround comment * test: check default config values in test_modeling_solar_open.py * fix: config class interface * docs: add SolarOpen to doctree * docs: update dates * Revert "feat: add solar_open to tokenizer mapping" This reverts commit 038b1c1. * feat: remove unnecessary configs * test: update SolarOpenConfig tests * fix: attention_dropout issue on training * Revert "feat: remove unnecessary configs" This reverts commit 9023688. * Revert "fix: attention_dropout issue on training" This reverts commit 3c275dc. * Revert "Revert "feat: remove unnecessary configs"" This reverts commit e6adcd9. * Revert "Revert "fix: attention_dropout issue on training"" This reverts commit 573fa9a. * feat: inherit attention from Llama * fix: remove del for attention_bias and attention_dropout * chore: convert solar_open * fix date --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> Co-authored-by: vasqu <antonprogamer@gmail.com>

oesni added 8 commits January 9, 2026 06:23

feat: implement solar-open-100b

596033b

feat: update modeling_solar_open.py

9110c56

feat: update solar-open config

1affcd2

chore: apply style

acdff18

feat: remove _tied_weights_keys

645a90c

feat: update modeling code

8a4dab7

chore: remove speech_to_text_2 in modeling

8c35387

docs: solar_open model

c147b71

yonigozlan added the New model label Jan 13, 2026

test: solar open model

42ec592

oesni changed the title ~~[WIP] Add Solar-Open Model~~ Add Solar-Open Model Jan 14, 2026

oesni marked this pull request as ready for review January 14, 2026 08:42

Merge branch 'main' into solar-open-100b

c1fc151

github-actions Bot requested review from ArthurZucker and Rocketknight1 January 14, 2026 08:42

chore: re-convert modular

455720d

fix: remove require_read_token

5126e11

vasqu reviewed Jan 14, 2026

View reviewed changes

oesni and others added 8 commits January 15, 2026 11:21

Apply suggestion from @vasqu

a89780b

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

chore: update lincse year -> 2026

5a8f862

feat: add solar_open to tokenizer mapping

038b1c1

chore: update license year

10038d4

test: remove _torch_compile_train_cls

f09cac5

docs: update solar_open doc

f80d1d5

refactor: simplify SolarOpenDecoderLayer

25d84b9

refactor: inherit Glm4MoeConfig class

e0487c3

vasqu approved these changes Jan 19, 2026

View reviewed changes

oesni and others added 12 commits January 20, 2026 15:07

feat: remove unnecessary configs

9023688

test: update SolarOpenConfig tests

583a8bf

fix: attention_dropout issue on training

3c275dc

Revert "feat: remove unnecessary configs"

e6adcd9

This reverts commit 9023688.

Revert "fix: attention_dropout issue on training"

573fa9a

This reverts commit 3c275dc.

Revert "Revert "feat: remove unnecessary configs""

5b6c6b2

This reverts commit e6adcd9.

Revert "Revert "fix: attention_dropout issue on training""

a5f4c69

This reverts commit 573fa9a.

feat: inherit attention from Llama

ad2d310

fix: remove del for attention_bias and attention_dropout

320891e

chore: convert solar_open

4d1c502

Merge branch 'main' into solar-open-100b

8888af4

fix date

1556e68

Merge branch 'main' into solar-open-100b

4a0f225

vasqu enabled auto-merge (squash) January 21, 2026 16:36

vasqu disabled auto-merge January 21, 2026 16:38

vasqu enabled auto-merge (squash) January 21, 2026 16:39

vasqu merged commit 93dd4fb into huggingface:main Jan 21, 2026
26 checks passed

Conversation

oesni commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

vasqu commented Jan 13, 2026

Uh oh!

oesni commented Jan 14, 2026

Uh oh!

oesni commented Jan 14, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu commented Jan 14, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vasqu commented Jan 19, 2026

Uh oh!

vasqu commented Jan 21, 2026

Uh oh!

github-actions Bot commented Jan 21, 2026

Uh oh!

github-actions Bot commented Jan 21, 2026

CI Results

Uh oh!

github-actions Bot commented Jan 21, 2026

Uh oh!

vasqu commented Jan 21, 2026

Uh oh!

github-actions Bot commented Jan 21, 2026

Uh oh!

github-actions Bot commented Jan 21, 2026

CI Results

Uh oh!

vasqu commented Jan 21, 2026

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jan 21, 2026

Uh oh!

oesni commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

oesni commented Jan 13, 2026 •

edited

Loading