Enable GRPO Qwen 3 32B with 128k context length

[wangshangsam]

Is your feature request related to a problem? Please describe.

Nemotron folks (i.e., @pjin-nvidia and @bxyu-nvidia ) would like to run GRPO experiments on Qwen 3 32B with 128k context length.

Currently they were able to launch the following configs:

• Qwen3 30B MoE, TP=4, 4x nodes, 32768 max context: https://wandb.ai/nvidia/pjin_nemo_rl_dev/runs/q9p0bvii/workspace?nw=nwuserpjinnvidia
• Qwen3 32B dense, TP=8, 8x nodes, 49152 max context: https://wandb.ai/nvidia/pjin_nemo_rl_dev/runs/n3nlkqgr/workspace?nw=nwuserpjinnvidia

For this task, there's no restrictions on:

• DTensor path vs. Megatron Core path. Anything works would be good.
• Runtime. It would be good if the experiments can finish overnight, but effectively, as of now, Nemotron folks are not blocked by how long the experiments take to finish.

The current understanding of the technical challenge is on GPU global memory footprint, since activations dominate the memory footprint, the size of which scales linearly with the sequence length.

Describe the solution you'd like

Being able to run GRPO Qwen 3 32B + 128k context length with a reasonable number of nodes.

• The initial target is 8 8xH100 nodes.

Describe alternatives you've considered

Additional context

[snowmanwwg]

This is needed for parallel reasoning. 16 x8.
Nemotron only needs 64k.

[ann-guan]

Why this was created and impactful: We are doing research on models before the next Ultra push. There are efforts to validate various verifiers, research parallel reasoning (like Grok 4 Heavy), improve multi turn and multi step agentic capabilities, and much more. We cannot meaningfully train any models on the parallel reasoning or multi turn/step agent front using NeMo RL.

• For the parallel reasoning push, for a single user query, 8 calls will be made to a model that may think up to 16k or more tokens. These 8 calls are then aggregated and sent to the model in a final model call for the final response to the user. This results in 8 * 16k = 128k tokens context length necessary for the final model call.
• For the multi turn/step agent push, a single agent trajectory may be dozens of steps. Including past reasoning traces (key for consistency) and tool responses over dozens of steps quickly adds up to 100k+ tokens of context. Even just 10 steps (medium horizon) at 16k reasoning tokens is 160k tokens necessary. We cannot even consider long horizon tasks that consist of e.g. 25+ steps.

For the near term, we understand that NeMo RL cannot support larger MoE training, but we should be able to support dense models up to 32B. We are additionally not asking for support for larger dense model sizes like Qwen 2.5 72B. That request is tracked in a separate issue below.
For the near term, we also understand that NeMo RL cannot support significantly longer sequence length training like 1M context length. For the interim, we are only asking for 128k context length.
Thank you to Qidong Su CA for pushing on this!