Use native collective instead of vllm's in non-colocated

[yuki-97]

Currently we use vllm.distributed.device_communicators.pynccl to broadcast and update weights in our non-colocated implementation (#489) since ray.util.collective can't work well when vllm tp-size>1 for now.

It will make our train worker coupled with specific inference backend, so it's better to find how to use a native ray collective to decouple them.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[sahgerlad]

Use Case: SFT-only Training Without vLLM

I encountered this coupling issue while trying to run SFT training only (no inference/generation) with the DTensor backend.

Problem

When using DTensorPolicyWorker for SFT, the worker requires the vLLM environment even though vLLM is not needed for training:

• ray_actor_environment_registry.py (lines 30-31)
• Temporary workaround for the coupled implementation of DTensorPolicyWorker and vLLM.
• This will be reverted to PY_EXECUTABLES.BASE once Use native collective instead of vllm's in non-colocated #501 is resolved.
  "nemo_rl.models.policy.dtensor_policy_worker.DTensorPolicyWorker": VLLM_EXECUTABLE,

This creates dependency conflicts when:

• Using newer PyTorch versions (e.g., 2.9+) that vLLM doesn't yet support
• Running on hardware where vLLM has compatibility issues
• Wanting a minimal SFT-only setup without inference dependencies

Workaround

For SFT-only workflows, changing to PY_EXECUTABLES.BASE works:
"nemo_rl.models.policy.dtensor_policy_worker.DTensorPolicyWorker": PY_EXECUTABLES.BASE,

Suggestion

Would it be possible to either:

1. Add a configuration option to select the executable environment for DTensorPolicyWorker
2. Or split the registry based on use case (training vs. inference)?

Happy to help test any proposed solutions!

[yuki-97]

Hi @sahgerlad , thanks for the interest in NeMo RL!

We suggest to use the WAR for your training at the moment. We had some discussions about it, and feel it'll make the structure of venvs more complicated and bloat the docker image.