fix: adjust temperature scaling logic based on engine version

[jubick1337]

What does this PR do ?

This PR fixes a training instability bug by conditionally applying temperature scaling to logits based on the active vLLM engine version.

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• This fix addresses an issue where logits were being scaled by temperature twice when using vLLM's V1 engine. The V1 engine returns raw, unscaled logits, while the V0 engine returned logits that were already scaled.

• The fix checks the VLLM_USE_V1 environment variable to determine if the V1 engine is active and only applies manual temperature division if it is not, ensuring the scaling is performed exactly once.

[parthchadha]

LGTM, thanks for the fix! One of our unit tests should have failed when v0=>v1 switch happened but I guess our unit tests were not robust enough.

[parthchadha]

@jubick1337 looks like lint check are failing, can you please run pre-commit as suggested at https://github.com/NVIDIA-NeMo/RL/blob/main/CONTRIBUTING.md.

[jubick1337]

Hi @parthchadha,

Thanks for your help. I fixed the linting.

Now mypy check is failing on files I did not change. What should I do next?

[terrykong]

Hi @jubick1337 . Thanks for helping make NeMo RL better!

The mypy job is expected to fail since we haven't 100% fixed our static typing issues, so you are a.o.k.

I've enqueued your PR