feat: Add DAPO dataset and Deepseek-v3 config

[yfw]

What does this PR do ?

This PR pulls in the DAPO dataset from #602 and also adds a config for running Deepseek-V3 on this dataset.

Closes #1142

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

Sample run with Deepseek-V3 training on DAPO-Math-17k and validation on AIME-2024:

Summary by CodeRabbit

• New Features

  • Added DAPO Math17K dataset support for training and validation.
  • Introduced optional DAPO math answer verification with boxed-answer and normalization logic.
  • Enabled GRPO training setups for DAPO Math17K with Megatron and vLLM generation, including precision, batching, and sequence packing options.

• Configuration

  • Provided end-to-end example configs covering optimizer/scheduler defaults, parallelism, and resource settings.
  • Included logging integrations (W&B, TensorBoard) and environment/cluster parameters.
  • Added dataset, generation, and validation scheduling controls, plus checkpointing options.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds two GRPO Megatron config recipes, introduces the DAPO Math 17K dataset loader/formatter, adds a DAPO math verifier module with normalization and scoring, and updates math environment verification to optionally use the new verifier via a configuration flag.

Changes

Cohort / File(s)	Summary
GRPO Megatron configs examples/configs/grpo_dapomath17k_1B_megatron.yaml, examples/configs/grpo_dapomath17k_dsv3_megatron.yaml	New end-to-end GRPO experiment YAMLs targeting Megatron, defining prompts, training/validation, policy/model, optimizer/scheduler, generation (vLLM), logging, and cluster settings.
Response dataset: DAPO Math 17K nemo_rl/data/datasets/response_datasets/__init__.py, nemo_rl/data/datasets/response_datasets/dapo_math.py	Adds DAPOMath17KDataset with formatting/mapping utilities and dataset preparation (train/validation). Exposes loader via all and integrates into dataset factory.
Math verifier (new) and environment integration nemo_rl/environments/dapo_math_verifier.py, nemo_rl/environments/math_environment.py	New verifier implementing boxed/minerva-style normalization and correctness checks; math environment now optionally uses it via use_dapo_math_verifier flag and updated HFVerifyWorker.verify signature.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Trainer
  participant Env as MathEnvironment
  participant Worker as HFVerifyWorker
  participant DapoV as dapo_math_verifier
  participant Legacy as verify_func

  Trainer->>Env: step(..., config.use_dapo_math_verifier)
  Env->>Worker: verify(pred_responses, ground_truths, return_extracted_answer, use_dapo_math_verifier)

  alt use_dapo_math_verifier = true
    Worker->>DapoV: compute_score(solution_str, ground_truth, strict_box_verify?)
    DapoV-->>Worker: {score, acc, pred_answer}
  else
    Worker->>Legacy: verify_func(pred, ground_truth_parsable)
    Legacy-->>Worker: score/extracted_answer
  end

  Worker-->>Env: scores, (optional) extracted answers
  Env-->>Trainer: step result

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes Check	⚠️ Warning	The inclusion of a standalone 1B-parameter Megatron GRPO configuration (grpo_dapomath17k_1B_megatron.yaml) goes beyond the stated scope of providing the Deepseek-V3 AIME GPRO recipe outlined in issue #1142 and is not referenced in the PR objectives.	Please remove or move the grpo_dapomath17k_1B_megatron.yaml file to a separate PR or document its intended use so that this PR remains focused on the DSV3 AIME GPRO end-to-end recipe.
Test Results For Major Changes	⚠️ Warning	The PR introduces substantial new functionality, including a new dataset integration, verification logic, and multiple training configurations, so it qualifies as a major change. I reviewed the PR description provided in the objectives and found no mention of executed tests, convergence validation, or performance measurements. Because no testing information is documented for these significant updates, the check cannot pass.	Please update the PR description with the relevant test results or validation evidence, such as dataset preprocessing checks, verifier correctness experiments, and any training or convergence runs for the new configurations, so this requirement can be satisfied.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title clearly and concisely describes the addition of the DAPO dataset and the Deepseek-v3 configuration, which are the primary features introduced by this pull request, without extraneous detail or noise.
Linked Issues Check	✅ Passed	This PR fully implements the end-to-end convergence recipe for the DSV3 AIME GPRO workflow by adding the DAPOMath17K dataset loader, integrating it into the response dataset registry, providing the Deepseek-V3 (DSV3) GRPO config, and updating the verification environment to support the new dataset, thus satisfying issue #1142.
Docstring Coverage	✅ Passed	Docstring coverage is 92.31% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch yifu/dapo_data

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

nemo_rl/environments/math_environment.py (1)

97-123: Fix extracted-answer handling for DAPO verifier.

When return_extracted_answer is True, the DAPO branch assigns reward_dict["pred"] (a plain string) to extracted_answer, so the later assertions (len(extracted_answer) == 2) and indexing (extracted_prediction[0][0]) crash. Wrap the DAPO prediction in the same (extracted_gold, extracted_prediction) structure returned by math_metric before entering the shared post‑processing.

Apply this diff:

-                    if use_dapo_math_verifier:
-                        # This compute_score is from the DAPO Math Verifier from Verl
-                        reward_dict = dapo_math_verify(response, ground_truth)
-                        ret_score = reward_dict["score"]
-                        extracted_answer = reward_dict["pred"]
+                    if use_dapo_math_verifier:
+                        # This compute_score is from the DAPO Math Verifier from Verl
+                        reward_dict = dapo_math_verify(response, ground_truth)
+                        ret_score = reward_dict["score"]
+                        extracted_pred = reward_dict["pred"]
+                        if return_extracted_answer:
+                            extracted_answer = (
+                                [ground_truth],
+                                [[extracted_pred]],
+                            )
+                        else:
+                            extracted_answer = None

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7536be3 and b9b461e.

📒 Files selected for processing (6)

• examples/configs/grpo_dapomath17k_1B_megatron.yaml (1 hunks)
• examples/configs/grpo_dapomath17k_dsv3_megatron.yaml (1 hunks)
• nemo_rl/data/datasets/response_datasets/__init__.py (3 hunks)
• nemo_rl/data/datasets/response_datasets/dapo_math.py (1 hunks)
• nemo_rl/environments/dapo_math_verifier.py (1 hunks)
• nemo_rl/environments/math_environment.py (4 hunks)

🧰 Additional context used

📓 Path-based instructions (3)

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Follow the Google Python Style Guide for all Python code
Target Python 3.12+ for all Python code in NeMo-RL
Indent Python code with 4 spaces; do not use tabs
Python filenames should be snake_case (e.g., some_file.py)
Class names should be PascalCase
Function and method names should be snake_case
Local variable names should be snake_case; if starting with a number, prefix with k (e.g., k_99th_percentile)
Global variables should be UPPER_SNAKE_CASE and prefixed with G_ (e.g., G_MY_GLOBAL)
Constants should be UPPER_SNAKE_CASE
Avoid shadowing variables declared in an outer scope
Initialize all externally visible members of a class in the constructor
For public interfaces used outside a file, prefer docstrings over comments
Use comments mainly for code within a function or interfaces local to a file
Commented-out code must include a nearby comment explaining usage and why it is commented out; otherwise remove before merging
Use Google-style docstrings for classes and functions (Sphinx-parseable)
Avoid using reflection when functionality can be easily achieved without it
Limit except clauses to the smallest specific set of exceptions possible
For duck-typing via try/except, keep the try body minimal and use else for main logic
Add the NVIDIA copyright header (with current year) at the top of all Python files, excluding tests/ and test-only scripts

Files:

• nemo_rl/environments/dapo_math_verifier.py
• nemo_rl/data/datasets/response_datasets/__init__.py
• nemo_rl/data/datasets/response_datasets/dapo_math.py
• nemo_rl/environments/math_environment.py

nemo_rl/**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

nemo_rl/**/*.py: Do not set non-None configuration defaults in code; YAML is the single source of truth for defaults
Access required config attributes directly (e.g., policy_cfg["precision"]) and assume presence; do not introduce hidden defaults
Express configuration optionality via TypedDict using typing.NotRequired
When adding a new config key to a TypedDict subclass, document the key’s purpose, valid values/types, and recommended default in code
For any class or function decorated with @ray.remote, add '# pragma: no cover' on the class/def line (and on remote functions)

Files:

• nemo_rl/environments/dapo_math_verifier.py
• nemo_rl/data/datasets/response_datasets/__init__.py
• nemo_rl/data/datasets/response_datasets/dapo_math.py
• nemo_rl/environments/math_environment.py

examples/configs/*.yaml

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

examples/configs/*.yaml: Exemplar configs under examples/configs/.yaml must include documented defaults
When adding a new config key, reflect its recommended default in exemplar YAMLs under examples/configs/.yaml

Files:

• examples/configs/grpo_dapomath17k_1B_megatron.yaml
• examples/configs/grpo_dapomath17k_dsv3_megatron.yaml

🧬 Code graph analysis (2)

nemo_rl/environments/dapo_math_verifier.py (1)

nemo_rl/environments/math_environment.py (3)

• verify (73-134)
• verify (139-179)
• verify (184-221)

nemo_rl/environments/math_environment.py (1)

nemo_rl/environments/dapo_math_verifier.py (1)

• compute_score (248-282)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: Coverage (e2e)
• GitHub Check: Coverage (doc-test)
• GitHub Check: Post automodel integration comment / Comment on PR
• GitHub Check: Post submodule check comment / Comment on PR

[terrykong]

generally lgtm, few nits

Could you add the convergence plots to the description?

also, i think it would be good to add this to the front page readme under "news" to make it clear that it's supported now

[ashors1]

LGTM from the dataset side. FYI @peri044 we'll have to rebase the DAPO PR on this one once it goes in