chore: major version bump (torch 2.8, vllm 0.11, ray 2.49) & SP fixes

[terrykong]

What does this PR do ?

Upgrades

Component	BASE	CANDIDATE
uv	0.7.2	0.9.1
torch	2.7.1	2.8.0
ray	2.46	2.49.2
vllm	0.10	0.11
flash-attn	0.2.7.4.post1	2.8.1
transformer-engine	2.5	2.8
cuda	12.8	12.9

Changes

• fixes some failing nightly configs
• disables SP and assert guard it
• remove some unnecessary patches
• add some sanity unit test for vllm sampling transforms

Issues

List issues that this PR closes (syntax):

Closes #1122 #818

closes #532

Will be broken once we upgrade:

• Qwen parallelizer with sequence parallelism Automodel#652 (automodel SP doesn't work)
• FP8 fix after vllm 0.11 upgrade #1368 FP8 will be broken
• llama nemotron broken after torch 2.8 upgrade #1374 llama-nemotron will be broken

Will be unblocked by this PR:

• feat: [1/2] Top-k and Top-p support for dtensor worker with vLLM V0 when TP==1 #773
• Support GRPO for Nemotron-H #1127

Tests

BASE	CANDIDATE	Test Name	Notes
PASS	PASS	distillation-qwen3-32b-to-1.7b-base-1n8g-fsdp2tp1.v1
PASS	PASS	distillation-qwen3-32b-to-4b-base-1n8g-fsdp2tp2-dynamicbatch.v1
PASS	PASS	distillation-qwen3-32b-to-4b-base-2n8g-fsdp2tp8-noncolocated.v1
PASS	PASS	dpo-llama3.1-8b-instruct-4n8g-fsdp2tp2-quick.v2
PASS	PASS	dpo-llama3.1-8b-instruct-4n8g-fsdp2tp4
PASS	PASS	dpo-llama3.1-8b-instruct-4n8g-megatrontp2pp2-quick
PASS	PASS	dpo-llama3.1-8b-instruct-4n8g-megatron.v2
PASS	PASS	dpo-llama3.1-8b-tulu3-1n8g-fsdp2tp1
PASS	PASS	dpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v2
PASS	PASS	grpo-deepscaler-1.5b-16K
PASS	PASS	grpo-deepscaler-1.5b-24K
PASS	PASS	grpo-deepscaler-1.5b-8K
PASS	PASS	grpo-gemma3-1b-it-1n8g-fsdp2tp1
PASS	PASS	grpo-gemma3-27b-it-8n8g-fsdp2tp8-actckpt-long
PASS	PASS	grpo-gspo-deepscaler-1.5b-8K
PASS	UNK	grpo-llama3.1-8b-instruct-1n8g-megatron-fp8-rollouts.v2	#1368
PASS	FAIL	grpo-llama3.1-8b-instruct-4n8g-fsdp2tp1-long.v3	random logprob spike
PASS	PASS	grpo-llama3.2-1b-instruct-1n8g-fsdp2tp1.v3
PASS	PASS	grpo-llama3.2-1b-instruct-1n8g-megatron
PASS	UNK	grpo-math-llama-nemotron-super-49b-v.5-4n8g-fsdp2tp8	#1374
PASS	PASS	grpo-math-qwen3-30ba3b-megatron-tp4-32k
PASS	PASS	grpo-moonlight-16ba3b-4n8g-megatron
PASS	PASS	grpo-qwen2.5-32b-32n8g-fsdp2tp8-actckpt-long.v3
PASS	PASS	grpo-qwen2.5-32b-32n8g-fsdp2tp8-actckpt.v3
PASS	PASS	grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3
FAIL	PASS	grpo-qwen2.5-7b-instruct-4n8g-megatron
PASS	PASS	grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3
PASS	PASS	grpo-qwen3-30ba3b-8n8g-megatron
FAIL	FAIL	sft-llama3.1-70b-8n8g-tp4pp2-long-megatron	both above mem thresh
PASS	PASS	sft-llama3.1-8b-1n8g-fsdp2tp1-dynamicbatch
PASS	PASS	sft-llama3.1-8b-1n8g-fsdp2tp1-long
PASS	PASS	sft-llama3.1-8b-1n8g-fsdp2tp2sp
PASS	PASS	sft-llama3.1-8b-1n8g-megatron
PASS	PASS	sft-llama3.1-8b-1n8g-megatron-seqpack
PASS	PASS	sft-llama3.2-1b-1n8g-fsdp2tp1.v3
PASS	PASS	sft-qwen2.5-32b-4n8g-fsdp2tp8sp-actckpt.v3
PASS	PASS	vlm_grpo-qwen2.5-vl-3b-instruct-clevr-1n2g-dtensor2tp1.v1
PASS	PASS	vlm_grpo-qwen2.5-vl-3b-instruct-clevr-1n2g-megatrontp2.v1

converge logs

big-version-bump.log

Summary by CodeRabbit

Release Notes

• New Features

  • Added sampling utility functions for enhanced logit filtering capabilities.
  • Improved vLLM integration with updated logprobs handling.

• Bug Fixes

  • Disabled sequence parallelism in multiple model configurations to address compatibility issues.
  • Added validation guards for tensor parallel configurations.

• Chores

  • Updated core dependencies: PyTorch 2.8.0, Ray 2.49.2, vLLM 0.11.0, flash-attn 2.8.1, transformer-engine 2.8.0.
  • Updated build infrastructure and CI/CD pipeline.
  • Enhanced memory management configurations across model recipes.

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 1378254 (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 64be4ba (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 6a5d8ce (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 5a16365 (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: bfa90c8 (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 79aa3bf (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: a87f3e9 (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[terrykong]

H100 + arm run looks sensible:

[github-actions]

ℹ️ File Consistency Check

Check based on commit: cb73810 (PR #1334 from tk/big-version-bump)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: cb73810 (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

ℹ️ File Consistency Check

Check based on commit: 92e34fe (PR #1334 from tk/big-version-bump)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 92e34fe (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

ℹ️ File Consistency Check

Check based on commit: 0886c0f (PR #1334 from tk/big-version-bump)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 0886c0f (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

ℹ️ File Consistency Check

Check based on commit: 604f51d (PR #1334 from tk/big-version-bump)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 604f51d (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: b9d8a3d (PR #1334 from tk/big-version-bump)

✅ Submodules that are properly updated:

Automodel: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

[youngeunkwon0405]

Hi @terrykong, I am having error in uv sync command after this PR. Do you have any idea of how to solve this issue?

youngeunk@cw-dfw-cs-001-vscode-01:/lustre/fs1/portfolios/coreai/users/youngeunk/mount/nemo-rl-youngeunk$ git submodule update --init --recursive 
From https://github.com/NVIDIA-NeMo/Automodel
 * branch            a2db048383cd54b3fafc928df4c30bf7bbf7c430 -> FETCH_HEAD
Submodule path '3rdparty/Automodel-workspace/Automodel': checked out 'a2db048383cd54b3fafc928df4c30bf7bbf7c430'
youngeunk@cw-dfw-cs-001-vscode-01:/lustre/fs1/portfolios/coreai/users/youngeunk/mount/nemo-rl-youngeunk$ uv sync
warning: Failed to parse `pyproject.toml` during settings discovery:
  TOML parse error at line 205, column 10
      |
  205 | [tool.uv.extra-build-dependencies]
      |          ^^^^^^^^^^^^^^^^^^^^^^^^
  unknown field `extra-build-dependencies`, expected one of `required-version`, `native-tls`, `offline`, `no-cache`, `cache-dir`, `preview`, `python-preference`, `python-downloads`, `concurrent-downloads`, `concurrent-builds`, `concurrent-installs`, `index`, `index-url`, `extra-index-url`, `no-index`, `find-links`, `index-strategy`, `keyring-provider`, `allow-insecure-host`, `resolution`, `prerelease`, `fork-strategy`, `dependency-metadata`, `config-settings`, `no-build-isolation`, `no-build-isolation-package`, `exclude-newer`, `link-mode`, `compile-bytecode`, `no-sources`, `upgrade`, `upgrade-package`, `reinstall`, `reinstall-package`, `no-build`, `no-build-package`, `no-binary`, `no-binary-package`, `python-install-mirror`, `pypy-install-mirror`, `python-downloads-json-url`, `publish-url`, `trusted-publishing`, `check-url`, `add-bounds`, `pip`, `cache-keys`, `override-dependencies`, `constraint-dependencies`, `build-constraint-dependencies`, `environments`, `required-environments`, `conflicts`, `workspace`, `sources`, `managed`, `package`, `default-groups`, `dependency-groups`, `dev-dependencies`, `build-backend`

⠼ Resolving dependencies...                                                                                                                                                                                                                                                                                                                                                                      × Failed to build `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)

      [stderr]
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
          self.run_setup()
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 512, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 7, in <module>
      ModuleNotFoundError: No module named 'torch'

      hint: This error likely indicates that `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef` depends on `torch`, but doesn't declare it as a build dependency. If `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef` is a first-party package, consider adding `torch` to its
      `build-system.requires`. Otherwise, `uv pip install torch` into the environment and re-run with `--no-build-isolation`.

[afennelly-mitre]

Hi @terrykong, I am having error in uv sync command after this PR. Do you have any idea of how to solve this issue?

youngeunk@cw-dfw-cs-001-vscode-01:/lustre/fs1/portfolios/coreai/users/youngeunk/mount/nemo-rl-youngeunk$ git submodule update --init --recursive 
From https://github.com/NVIDIA-NeMo/Automodel
 * branch            a2db048383cd54b3fafc928df4c30bf7bbf7c430 -> FETCH_HEAD
Submodule path '3rdparty/Automodel-workspace/Automodel': checked out 'a2db048383cd54b3fafc928df4c30bf7bbf7c430'
youngeunk@cw-dfw-cs-001-vscode-01:/lustre/fs1/portfolios/coreai/users/youngeunk/mount/nemo-rl-youngeunk$ uv sync
warning: Failed to parse `pyproject.toml` during settings discovery:
  TOML parse error at line 205, column 10
      |
  205 | [tool.uv.extra-build-dependencies]
      |          ^^^^^^^^^^^^^^^^^^^^^^^^
  unknown field `extra-build-dependencies`, expected one of `required-version`, `native-tls`, `offline`, `no-cache`, `cache-dir`, `preview`, `python-preference`, `python-downloads`, `concurrent-downloads`, `concurrent-builds`, `concurrent-installs`, `index`, `index-url`, `extra-index-url`, `no-index`, `find-links`, `index-strategy`, `keyring-provider`, `allow-insecure-host`, `resolution`, `prerelease`, `fork-strategy`, `dependency-metadata`, `config-settings`, `no-build-isolation`, `no-build-isolation-package`, `exclude-newer`, `link-mode`, `compile-bytecode`, `no-sources`, `upgrade`, `upgrade-package`, `reinstall`, `reinstall-package`, `no-build`, `no-build-package`, `no-binary`, `no-binary-package`, `python-install-mirror`, `pypy-install-mirror`, `python-downloads-json-url`, `publish-url`, `trusted-publishing`, `check-url`, `add-bounds`, `pip`, `cache-keys`, `override-dependencies`, `constraint-dependencies`, `build-constraint-dependencies`, `environments`, `required-environments`, `conflicts`, `workspace`, `sources`, `managed`, `package`, `default-groups`, `dependency-groups`, `dev-dependencies`, `build-backend`

⠼ Resolving dependencies...                                                                                                                                                                                                                                                                                                                                                                      × Failed to build `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)

      [stderr]
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
          self.run_setup()
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 512, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/youngeunk/.cache/uv/builds-v0/.tmpKu42nX/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 7, in <module>
      ModuleNotFoundError: No module named 'torch'

      hint: This error likely indicates that `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef` depends on `torch`, but doesn't declare it as a build dependency. If `deep-ep @ git+https://github.com/deepseek-ai/DeepEP.git@e3908bf5bd0cc6265bcb225d15cd8c996d4759ef` is a first-party package, consider adding `torch` to its
      `build-system.requires`. Otherwise, `uv pip install torch` into the environment and re-run with `--no-build-isolation`.

I am also experiencing several issues when trying to get my python environment setup after the changes introduced in this PR!!

[terrykong]

@youngeunkwon0405 @afennelly-mitre In both of your cases it's due to uv needing to be upgraded. Please update to 0.9.1

Just a heads up that I am aware that this change also made the dependency resolution tempermental. If you run into an error like this for causal-conv1d or any of the ones with extra-build-denepdencies:

× Failed to download and build `deep-ep @
  │ git+https://github.com/deepseek-ai/DeepEP.git`
  ╰─▶ Extra build requirement `torch` was declared with `match-runtime
      = true`, but `deep-ep` does not declare static metadata, making
      runtime-matching impossible

I believe the fix is this: d7a3e40 (see only the change to the [[tool.uv.dependency-metadata]]. It seems that without this changing dependencies un-related to these will trigger a failure. This change will land as part of #1358, but i'm testing it right now

[afennelly-mitre]

@youngeunkwon0405 @afennelly-mitre In both of your cases it's due to uv needing to be upgraded. Please update to 0.9.1

Just a heads up that I am aware that this change also made the dependency resolution tempermental. If you run into an error like this for causal-conv1d or any of the ones with extra-build-denepdencies:

× Failed to download and build `deep-ep @
  │ git+https://github.com/deepseek-ai/DeepEP.git`
  ╰─▶ Extra build requirement `torch` was declared with `match-runtime
      = true`, but `deep-ep` does not declare static metadata, making
      runtime-matching impossible

I believe the fix is this: d7a3e40 (see only the change to the [[tool.uv.dependency-metadata]]. It seems that without this changing dependencies un-related to these will trigger a failure. This change will land as part of #1358, but i'm testing it right now

@terrykong what cuda version was used with the provided major version updates to things like torch and vllm? I'm having different issues when using cuda 12.5 and cuda 12.6

Trying to figure out if it's simply related to me needing a higher cuda version, or fixing something else with my environment setup.

[terrykong]

@afennelly-mitre our docker image uses 12.9. Local/bare-metal it's hard to support a large version skew, so I can imagine you might run into issues if the version is too far off or not an LTS version. I would recommend upgrading to 12.9, or if that is prohibitive on your workstation, I would encourage to use the container.