Widen transformers for v5.6 and vllm==0.19.1#1561
Open
jamesbraza wants to merge 9 commits intoNovaSky-AI:mainfrom
Open
Widen transformers for v5.6 and vllm==0.19.1#1561jamesbraza wants to merge 9 commits intoNovaSky-AI:mainfrom
transformers for v5.6 and vllm==0.19.1#1561jamesbraza wants to merge 9 commits intoNovaSky-AI:mainfrom
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
jamesbraza
force-pushed
the
upgrade-transformers-5-6
branch
6 times, most recently
from
e21d95b to
63734cf
Compare
jamesbraza
force-pushed
the
upgrade-transformers-5-6
branch
2 times, most recently
from
08793c0 to
ec09230
Compare
jamesbraza
changed the title
transformers for v5.6.0 and vllm==0.19.1transformers for v5.6 and vllm==0.19.1
jamesbraza
force-pushed
the
upgrade-transformers-5-6
branch
from
ec09230 to
24cfa2d
Compare
Set LoRA/sharding kwargs on self before super().__init__ so transformers 5.4.0's @strict validators (one of which calls self.get_text_config) can read them. Then raise if config.__dict__ carries any of those kwarg names, since super() would otherwise silently overwrite them via its setattr loop. The overlap check compares self.__dict__ against config.__dict__ so the kwarg names are not duplicated. Inherit attribute_map from the source config class so aliases like Qwen3MoeConfig's `num_experts` -> `num_local_experts` keep working after wrapping. Set on self (instance-scoped) rather than type(self) to avoid cross-instance leakage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ained
The base PretrainedConfig class-method path fails under transformers 5.5
for model configs whose rope_type is in {llama3, yarn, longrope}. Those
configs go through modeling_rope_utils.standardize_rope_params which reads
self.max_position_embeddings directly during __post_init__, before the
kwargs setattr loop runs; on base PreTrainedConfig the attribute doesn't
exist as a default and the read raises AttributeError.
AutoConfig.from_pretrained returns the model-specific subclass (LlamaConfig,
DeepseekV3Config, etc.) which declares max_position_embeddings as a field
with a default, so the un-guarded read succeeds. qwen3-family configs do
not trigger this code path but are swapped too for consistency and to
remove a latent foot-gun for any future llama3/yarn/longrope use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
transformers 5.4.0 added a @strict validator validate_layer_type that requires len(layer_types) == num_hidden_layers. The existing test helper shrinks num_hidden_layers to 1 for speed but leaves the original layer_types list from Qwen/Qwen3-0.6B (28 entries), which now raises StrictDataclassClassValidationError during config construction. Trim layer_types to match the new num_hidden_layers so the validator is satisfied. Guarded by getattr so it's a no-op for configs that don't define layer_types. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Gemma4Config (added in huggingface/transformers#45192) and other composite VLM configs (e.g., Qwen2.5-VL) nest attention fields under text_config rather than exposing them on the top-level config. The ulysses monkey patch read model.config.num_attention_heads directly, which raises AttributeError for these models. PreTrainedConfig.get_text_config returns self for text-only models and the text sub-config for VLMs, so this is a no-op for Qwen3/Llama3/DeepSeek and unblocks Gemma4 in transformers 5.6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduced as a misnomer in NovaSky-AI#889; DeepSeek V3 uses rope_type "yarn", as registered in ROPE_INIT_FUNCTIONS upstream: https://github.com/huggingface/transformers/blob/v5.6.0/src/transformers/modeling_rope_utils.py#L635 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The split invocation with `--with transformers==5.2.0` was added in NovaSky-AI#1228 when pyproject.toml still pinned transformers <5, to let the new Qwen 3.5 test use v5 while the rest of the suite stayed on v4. The project-wide migration to v5 in NovaSky-AI#1426 left this carve-out and its comment behind, so test_qwen3_5.py has been artificially pinned to 5.2.0 while everything else runs on whatever pyproject.toml resolves (now 5.5.4 on this branch). Collapse back to a single pytest invocation — the exact shape the workflow had before NovaSky-AI#1228 — so all tests run on one transformers version. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
transformers v5.5 deprecated the torch_dtype kwarg on from_pretrained in favor of dtype. Both still work but the old name now emits warning_once. Rename to silence the warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)
on transformers>=5.4.0 returns bfloat16 hidden_states because the `dtype`
keyword argument is consumed by AutoConfig and not forwarded to the model
when the config has a nested `text_config` — and Qwen3.5 does — per
huggingface/transformers#41250. Weights then load
from the checkpoint in bfloat16 and the model runs that way end-to-end,
diverging from the fp32 JAX model by ~6% at the final hidden state on
the 0.8B checkpoint — too much even for the prior rtol=2e-2.
Drop the silently-ignored `dtype=torch.float32` kwarg and chain `.float()`
on the loaded model so the HF reference actually runs in fp32. Also
loosen the final-hidden-state tolerance to rtol=atol=2e-2: the .float()
cast restores fp32 inference, but Qwen3.5's gated-delta-rule layers
(exp/cumsum/tril) still accumulate more JAX-vs-PyTorch fp32 rounding
across the stack than plain attention does, and the CI outlier exceeds
1e-2 even though local runs fit a tighter bound. The final assertion now
also prints the worst-element signed diff on failure so future drift is
diagnosable without a local repro. Earlier-layer assertions stay at
their original tolerances.
No matching change is needed in test_qwen3.py: Qwen3Config has no nested
text_config (get_text_config() returns self), so the `dtype=torch.float32`
kwarg is still forwarded to the model there, and that path is actually
preferable to .float()-after-load because it allocates weights directly
in fp32 instead of loading bf16 and up-casting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jamesbraza
force-pushed
the
upgrade-transformers-5-6
branch
from
24cfa2d to
d117ffa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1509