cp: Support DAPO dynamic sampling and reward shaping (#602) into r0.4.0

[chtruong814]

What does this PR do ?

cp: feat: Support DAPO dynamic sampling and reward shaping (#602) into r0.4.0

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

Summary by CodeRabbit

Release Notes

• New Features

  • Added DAPO (Dynamic Sampling Policy Optimization) with dynamic sampling, reward scaling, and reward shaping capabilities for improved policy training.
  • Introduced multiple math verification backends for flexible reward evaluation.
  • Added HuggingFace model configuration override support for customized model loading.

• Documentation

  • New comprehensive DAPO guide with quickstart, configuration examples, and algorithm details.

• Configuration

  • New example configurations for DAPO training with Qwen2.5-7B model and updated GRPO configs.

[github-actions]

ℹ️ File Consistency Check

Check based on commit: 4b1f07f (PR #1458 from chtruong/cp-602-r0.4.0)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[github-actions]

ℹ️ File Consistency Check

Check based on commit: d7a3586 (PR #1458 from chtruong/cp-602-r0.4.0)

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/dtensor_policy_worker.py
• nemo_rl/models/policy/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[coderabbitai]

Caution

Review failed

The head commit changed during the review from 4b1f07f to d7a3586.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch chtruong/cp-602-r0.4.0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}