Add settings to make training results deterministic

[redfler]

Description

This PR adds a deterministic training path and documentation for Isaac Lab RL workflows.

• Added apps/isaaclab.python.headless.determinism.kit as a deterministic headless rendering experience.
• Updated scripts/reinforcement_learning/rl_games/train.py to add opt-in --deterministic and use configure_seed(env_cfg.seed, args_cli.deterministic).
• Updated docs/source/features/reproducibility.rst to document --experience isaaclab.python.headless.determinism.kit
  and clarify that strict PyTorch determinism is currently exposed only for RL-Games.

Test command example:

./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py --task Isaac-Cartpole-RGB-v0 --enable_cameras --headless --seed 42 --max_iteration 20 --deterministic --experience isaaclab.python.headless.determinism.kit

Fixes # (issue)

#3505 Non-reproducible training results in vision-based tasks with identical seeds

Type of change

• Bug fix (non-breaking change which fixes an issue)

Screenshots

Before	After

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[isaaclab-review-bot]

🤖 Isaac Lab Review Bot

Summary

This PR adds determinism support for vision-based RL training by: (1) calling configure_seed() in the rl_games training script to enable PyTorch deterministic algorithms, and (2) providing a new .kit experience file with RTX renderer settings that force synchronous rendering. The changes are minimal and focused, addressing a known reproducibility issue (GitHub #3505) for camera-based tasks.

Architecture Impact

• train.py: The configure_seed() call affects only this specific training script. Other RL library wrappers (RSL-RL, SB3, SKRL) are NOT updated — users of those frameworks will still have non-deterministic vision training.
• .kit file: New experience file extends isaaclab.python.headless with RTX determinism settings. No breaking changes to existing .kit files.

Implementation Verdict

Minor fixes needed — The core approach is sound, but there are gaps in test coverage, documentation, and consistency across training scripts.

Test Coverage

❌ No tests added. The PR description acknowledges this (checklist unchecked). For a bug fix claiming to resolve reproducibility issues, a regression test demonstrating determinism would be valuable — e.g., running 2 training iterations with the same seed and asserting identical loss/reward values. Without tests, the claim in the screenshots cannot be automatically verified.

CI Status

No CI checks available yet — cannot verify the changes don't break existing functionality.

Findings

🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:207 — configure_seed hardcodes True for deterministic mode

configure_seed(env_cfg.seed, True)

The second argument enables torch.use_deterministic_algorithms(True), which can cause runtime errors for operations without deterministic implementations (e.g., certain scatter/gather ops on CUDA). This should either:

1. Be gated behind a CLI flag (e.g., --deterministic) so users can opt-in
2. At minimum, document which operations may fail

Looking at isaaclab.utils.seed.configure_seed, if the user hasn't explicitly requested determinism, forcing it could break existing workflows that previously worked.

🟡 Warning: Inconsistency across training scripts — only rl_games is updated
The RSL-RL (scripts/reinforcement_learning/rsl_rl/train.py), Stable-Baselines3, and SKRL training scripts are not updated. Users expecting determinism across all frameworks will be surprised. Either:

1. Update all training scripts consistently
2. Document that determinism is only supported for rl_games

🔵 Improvement: apps/isaaclab.python.headless.determinism.kit:168 — Missing newline at end of file

rtx.rtpt.lightcache.cached.enabled = false

File doesn't end with a newline. While minor, this violates POSIX standards and can cause issues with some tools.

🔵 Improvement: apps/isaaclab.python.headless.determinism.kit:12 — Package title doesn't reflect determinism purpose

title = "Isaac Lab Python Headless Camera"

The title says "Headless Camera" but this file is specifically for deterministic rendering. Should be "Isaac Lab Python Headless Deterministic Rendering" or similar for clarity.

🔵 Improvement: apps/isaaclab.python.headless.determinism.kit — Missing documentation comments explaining the key determinism settings
The critical settings for determinism are:

• Line 79: omni.replicator.asyncRendering = false
• Line 82-84: app.renderer.waitIdle=true, app.hydraEngine.waitIdle=true
• Line 165-167: rtx.rtpt.cached.enabled = false, rtx.rtpt.lightcache.cached.enabled = false

These should have comments explaining WHY they enable determinism, not just performance implications. Users looking at this file need to understand what makes it different from the non-deterministic version.

🔵 Improvement: PR description shows command example but no documentation update
The command example in the PR description:

./isaaclab.sh -p ... --experience isaaclab.python.headless.determinism.kit

This should be documented in the official docs (the checklist shows documentation is unchecked). Users won't discover this without documentation.

🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:207 — configure_seed called after environment creation
The configure_seed call happens after gym.make() and wrapper creation (lines 161-189). If environment initialization involves any non-deterministic operations (random initialization, etc.), those won't be affected by this late seeding. The seed should be configured before environment creation, not after. Looking at the code flow:

• Line 111: env_cfg.seed = agent_cfg["params"]["seed"] — seed is set in config
• Line 161: env = gym.make(...) — environment created
• Line 207: configure_seed(...) — deterministic mode enabled too late

This ordering means the environment creation itself may still be non-deterministic even with this fix.

[greptile-apps]

Greptile Summary

This PR adds two changes to make Isaac Lab RL training results deterministic for vision-based tasks: a new .kit experience file that enables RTX path-tracing determinism settings, and a configure_seed call in the rl_games train script to enable PyTorch deterministic mode.

• P1 — torch.use_deterministic_algorithms(True) unconditionally applied: configure_seed(env_cfg.seed, True) hardcodes torch_deterministic=True for every rl_games training run, not just deterministic/vision ones. This risks RuntimeError on any non-deterministic op and adds overhead without a user-facing opt-in flag. The other RL framework training scripts (rsl_rl, sb3, skrl) are unaffected, creating an inconsistency.
• The determinism .kit file is a verbatim copy of isaaclab.python.headless.rendering.kit with only 3 appended lines, which will require manual maintenance to stay in sync.

Confidence Score: 3/5

Not safe to merge as-is; hardcoded torch_deterministic=True is a breaking behavioural change for all rl_games training users.

A P1 finding (unconditional torch.use_deterministic_algorithms(True)) means any rl_games training run using non-deterministic PyTorch ops will fail with a RuntimeError after this change, and all other runs will silently incur determinism overhead. The fix is straightforward, but the current state is not merge-ready.

scripts/reinforcement_learning/rl_games/train.py — the hardcoded torch_deterministic=True argument needs to be made opt-in.

Important Files Changed

Filename	Overview
scripts/reinforcement_learning/rl_games/train.py	Adds configure_seed(env_cfg.seed, True) before runner.load(), but hardcodes torch_deterministic=True for every training run, which could break any rl_games training that uses non-deterministic PyTorch ops.
apps/isaaclab.python.headless.determinism.kit	New kit file for deterministic RTX rendering; effectively a full copy of isaaclab.python.headless.rendering.kit plus 3 path-tracing settings. Package title is stale (still says "Camera"), and file lacks a trailing newline.
CONTRIBUTORS.md	Adds author "Xu Li" in alphabetical order — no issues.

Sequence Diagram

sequenceDiagram
    participant User
    participant train.py
    participant gym as gym.make (env)
    participant configure_seed as configure_seed()
    participant runner as rl_games Runner

    User->>train.py: run with --seed 42 --experience determinism.kit
    train.py->>train.py: set env_cfg.seed from agent_cfg["params"]["seed"]
    train.py->>gym: gym.make(task, cfg=env_cfg) → env internally calls configure_seed(seed)
    gym-->>train.py: env created (seed already applied)
    train.py->>configure_seed: configure_seed(env_cfg.seed, True)
    Note over configure_seed: Sets torch.use_deterministic_algorithms(True)<br/>Sets CUBLAS_WORKSPACE_CONFIG<br/>Re-seeds torch/numpy/random/warp
    configure_seed-->>train.py: seed applied (deterministic mode ON)
    train.py->>runner: runner.load(agent_cfg)
    train.py->>runner: runner.run(train=True)
    Note over runner: Any non-deterministic torch op<br/>raises RuntimeError here

Loading

_{Reviews (1): Last reviewed commit: "added name to the CONTRIBUTORS.md" | Re-trigger Greptile}

[isaaclab-review-bot]

🤖 Isaac Lab Review Bot

Summary

This follow-up addresses the previous review findings. The author has made several improvements: added --deterministic CLI flag (addressing the hardcoded True concern), renamed the kit file title to reflect determinism, added documentation to reproducibility.rst, and added explanatory comments for determinism settings in the kit file.

Previous Concerns Addressed

✅ configure_seed now uses args_cli.deterministic instead of hardcoded True
✅ Kit file title updated to "Isaac Lab Python Headless Deterministic Rendering"
✅ Documentation added explaining the --experience flag and --deterministic option
✅ Comments added explaining RTPT cache settings for determinism

Remaining/New Findings

🟡 Warning: scripts/reinforcement_learning/rl_games/train.py:215 — configure_seed still called after environment creation
The previous concern about ordering remains unaddressed. configure_seed() is called at line 215, but gym.make() happens at line 161. Any randomness during environment initialization won't be affected by the deterministic settings. The call should be moved before env = gym.make(...).

🟡 Warning: Test coverage still missing
The PR checklist still shows tests are not added. For a bug fix PR claiming to resolve reproducibility issues (#3505), a regression test demonstrating determinism would strengthen confidence in the fix.

🔵 Improvement: apps/isaaclab.python.headless.determinism.kit — File still missing trailing newline
Minor POSIX compliance issue remains.

Implementation Verdict

Minor fixes needed — The seed ordering issue is the main remaining concern that could affect correctness.

[isaaclab-review-bot]

Follow-up Review

Thanks for addressing the previous feedback! The changes look good:

• ✅ Added explanatory comment for configure_seed ordering after Runner()
• ✅ Added determinism-critical section header in .kit file
• ✅ Added warning when --deterministic is used without compatible flags
• ✅ Increased max_iterations to 50 for more robust test assertions

Minor Polish (non-blocking)

Warning message formatting (app_launcher.py:1036-1038):

"--deterministic has no effect when not in headless mode or "
"when cameras are disabled or  when livestreaming or XR is enabled."
"Use --enable_cameras --headless --deterministic for deterministic rendering."

Two small issues:

1. Double space: "or when" (extra space)
2. Missing space before "Use" (strings concatenate directly, so "enabled." + "Use" = "enabled.Use")

Suggested fix:

"--deterministic has no effect when not in headless mode, "
"when cameras are disabled, or when livestreaming/XR is enabled. "
"Use --enable_cameras --headless --deterministic for deterministic rendering."

---

Overall the PR is well-structured and addresses a real reproducibility pain point. The approach of separating rendering determinism (kit experience file) from PyTorch determinism (configure_seed) is clean, and the test coverage is thoughtful.

Status: Approved pending minor string fix (or can merge as-is if team prefers).

---

Update (d2631d8): Excellent extension! This commit expands deterministic seed support to all RL frameworks:

Training scripts now support --deterministic:

• rsl_rl/train.py
• skrl/train.py
• sb3/train.py
• (rl_games already had it, now refactored to conditional check)

Play scripts also support --deterministic:

• rl_games/play.py
• rsl_rl/play.py
• skrl/play.py
• sb3/play.py

Test coverage expanded:

• New test_play_scripts_call_configure_seed_after_runner_or_agent_construction() for all play scripts
• Extended train script test to cover all RL frameworks (was only rl_games)

The pattern is consistent: configure_seed is always called after runner/agent construction to avoid interfering with internal initialization. The conditional if args_cli.deterministic check is cleaner than passing the flag through.

Previous approval stands. This is now a comprehensive determinism solution across the entire RL stack. 🎉

---

Update (8cc5a12): Rebase onto develop branch. The deterministic feature code is unchanged — the only modification touching RL scripts is a version bump in rsl_rl/train.py (RSL_RL_VERSION = "3.0.1" → "5.0.1"), which is unrelated to determinism.

✅ All previous approvals still valid. Ready to merge.

---

Update (aedb643): Rebase onto latest develop. The determinism feature code remains unchanged — this commit incorporates upstream changes (LEAPP export refactor, camera math fixes, debugging improvements) that don't affect the PR's core functionality.

✅ All previous approvals remain valid. Ready to merge.

[kellyguo11]

My feeling is we don't need to tie the deterministic flag towards a specific RL library, especially since rl-games is less maintained now and we are shifting away from it. Could we do the same setup that enables determinism for RL Games in the train and play scripts for other RL libraries as well?

[kellyguo11]

@StafaH for viz, this may impact your training script refactoring

[isaaclab-review-bot]

🔄 Follow-up on new commits (656255d)

Reviewed incremental changes since last review (159eb2b):

File	Change	Status
.github/workflows/config.yaml	Pinned isaacsim_image_tag to specific SHA digest	✅ Good for CI reproducibility
tools/test_settings.py	Increased test_articulation.py timeout 1500→3000	✅ Prevents flaky timeouts

No new issues found. These are CI/infrastructure improvements that complement the determinism features.

Previous approval stands — ready to merge (optional minor string fix noted in prior review).