OMPE-88188: Enable frame stacking for newton+newton_renderer by default

[jmart-nv]

Description

This enables frame stacking for newton+warp by default.

Previously this PR also included changes to RTX's default AA, which have been reverted. This was intended to remove the implicit temporal information introduced by DLSS and TAA. Instead, we now focus on explicitly adding temporal data via frame stacking where needed.

Newton Frame Stacking: Provides explicit temporal data to newton.

RTX uses DLSS anti-aliasing by default, which provides implicit temporal data. The newton_renderer does not provide temporal data. However, newton's energy-conserving physics solver requires temporal velocity data in order to compensate for the lack of damping, which causes convergence problems when paired with newton_renderer.

This commit provides explicit temporal information via 2-frame stacking by default for camera tasks that use newton+newton_renderer. This allows newton to provide the damping it needs to converge at the same rate as physx. This adds 36% GPU memory overhead, but the wall clock overhead is negligible.

The default for all other physics/renderer backends is still stack size = 1 (disabled) since physx has implicit damping built-in via its TGS solver, and RTX provides temporal data implicitly.

Implementation:

CameraCfg now has a frame_stack field that controls a ring buffer of previous frames. These are concatenated to the channel dimension automatically in DirectRLEnv. The new MultiBackendCameraCfg wraps the MultiBackendRendererCfg to provide defaults for the different presets, driven by nested config classes and capability flags. Warnings will be logged if it detects that a physics backend that needs explicit temporal data is paired with a renderer that does not provide it.

Updated existing tasks to use the new MultiBackendCameraCfg and added 65 new unit tests.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Screenshots

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings (N/A - new warnings are intentional)
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[kellyguo11]

Review Summary

Good two-part PR that addresses a real sim2real gap: (1) disabling temporal AA (DLSS) by default so RL policies don't learn from renderer-specific temporal artifacts, and (2) adding frame stacking for Newton to explicitly provide temporal information that Newton's energy-conserving integrator needs.

The implementation is solid — ring buffer approach for frame stacking is clean, the MultiBackendTiledCameraCfg preset pattern is a nice API improvement over the previous renderer_cfg=MultiBackendRendererCfg() approach, and the test coverage is thorough (12+ new tests covering frame stacking, ring buffer correctness, reset behavior, and AA mode warnings).

A few performance and correctness issues worth addressing below.

[greptile-apps]

Greptile Summary

This PR adds an explicit frame-stacking ring buffer to CameraCfg/Camera and wires a two-level preset policy (MultiBackendCameraCfg + FrameStackPolicyCfg) that automatically enables frame_stack=2 for the Newton+Warp backend combination, while emitting a warning when physics requires temporal data and no stacking or temporal-AA renderer is present.

• Frame-stacking in Camera: _create_buffers pre-allocates _frame_history and _stacked_output once; each step advances the ring buffer using narrow()+copy_() with no per-step allocations, and resets correctly preserve per-env history isolation.
• Policy infrastructure: MultiBackendCameraCfg carries a sentinel frame_stack=0; launch_simulation calls _apply_frame_stack_policies to resolve the nested FrameStackPolicyCfg tree before scanning for the temporal-data warning, ensuring post-policy state is what the warning reflects.
• DirectRLEnv obs-space adjustment: A shallow scan of cfg.__dataclass_fields__ auto-multiplies the channel dimension for a 3-tuple observation_space, raising ValueError on conflicting stacks across multiple cameras.

Confidence Score: 4/5

Safe to merge; ring-buffer logic, reset isolation, sentinel handling, and the warning predicate matrix are all correct, with one minor documentation inaccuracy in camera_cfg.py.

The ring-buffer implementation deliberately avoids per-step allocations via narrow()+copy_(), but the docstring still says 'O(frame_stack) allocations' — directly contradicting the inline code comment at the call site. All functional logic is sound: history initialization on reset fills all slots with the first post-reset frame, partial resets isolate the correct env indices, the stacked output is built oldest-to-newest as expected, and the _apply_frame_stack_policies sentinel design cleanly separates user-supplied values from policy-driven defaults.

source/isaaclab/isaaclab/sensors/camera/camera_cfg.py has the documentation inaccuracy. The rest of the changed files look correct.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/sensors/camera/camera.py	Core ring-buffer implementation for frame stacking: correct initialization, reset, and per-step copy logic using narrow() views to avoid allocations
source/isaaclab/isaaclab/sensors/camera/camera_cfg.py	Adds frame_stack field with default=1; docstring claims O(frame_stack) allocations per step but implementation uses allocation-free narrow()+copy_()
source/isaaclab/isaaclab/envs/direct_rl_env.py	Adds obs-space auto-adjustment for frame stacking with conflict detection; shallow one-level scan is consistent with the 3-tuple single-camera convention
source/isaaclab_tasks/isaaclab_tasks/utils/sim_launcher.py	Adds _apply_frame_stack_policies, _resolve_frame_stack_policy, and the warning logic; traversal and sentinel handling look correct
source/isaaclab_tasks/isaaclab_tasks/utils/presets.py	Adds MultiBackendCameraCfg with sentinel frame_stack=0 and nested FrameStackPolicyCfg tree; two-level PresetCfg design for AND-conditioned physics×renderer selection is well-structured
source/isaaclab/test/sensors/test_camera_frame_stack.py	835-line test suite covering ring-buffer shape, init, reset, partial reset, renderer capability flags, warning matrix, policy truth table, CLI overrides, and end-to-end propagation
source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer_cfg.py	Adds provides_temporal_camera_data override returning False for pure-rasterization Warp renderer
source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer_cfg.py	Adds provides_temporal_camera_data returning True for DLSS/DLAA/TAA and False for FXAA/Off; correctly falls back to True when antialiasing_mode is None (DLSS default)

Sequence Diagram

sequenceDiagram
    participant CLI as CLI / resolve_task_config
    participant LS as launch_simulation
    participant AFP as _apply_frame_stack_policies
    participant DRLE as DirectRLEnv.__init__
    participant Cam as Camera._create_buffers

    CLI->>LS: "env_cfg (frame_stack=0 sentinel)"
    LS->>AFP: "walk cfg tree, set frame_stack=2 where newton+warp policy fires"
    AFP-->>LS: frame_stack resolved on MultiBackendCameraCfg nodes
    LS->>LS: _scan_config for needs_temporal / provides_temporal / has_unstacked_camera
    alt needs_temporal AND NOT provides_temporal AND has_unstacked_camera
        LS->>LS: logger.warning(frame_stack recommendation)
    end
    LS->>DRLE: env_cfg with frame_stack already resolved
    DRLE->>DRLE: "scan cfg.__dataclass_fields__ for frame_stack > 1"
    DRLE->>DRLE: "auto-adjust observation_space (H,W,C) to (H,W,C*frame_stack)"
    DRLE->>Cam: "Camera(cfg) with frame_stack=2"
    Cam->>Cam: alloc _frame_history shape (stack, N, H, W, C)
    Cam->>Cam: "alloc _stacked_output shape (N, H, W, C*stack)"
    Cam->>Cam: "_data.output = _stacked_output, renderer writes to _single_frame_output"
    loop each step
        Cam->>Cam: renderer writes new frame to _single_frame_output
        Cam->>Cam: history[idx].copy_(single), then narrow+copy_ to _stacked_output
    end

Loading

_{Reviews (1): Last reviewed commit: "OMPE-88188: Fix precommit errors." | Re-trigger Greptile}

[kellyguo11]

I don't think this should belong in the base class. the stacking mechanism should be on a per-task implementation. I would add this specifically for the cartpole environment

[kellyguo11]

I think this probably also doesn't belong in the physics manager cfg. there's no reason the physics manager should care about camera data

[kellyguo11]

I would recommend handling the frame stacking logic in the cartpole environment directly

[kellyguo11]

these are probably also not necessary if we move the logic into the task class directly?

[ooctipus]

Good Job figuring this out @jmart-nv I 100% agree with the motivation and surprised by your finding. Implementation wise, similar to Kelly's advise, I’d prefer this stay task-local for now. The current PR makes frame stacking a framework-level concern across the camera sensor, base env, physics config, renderer capabilities, presets, and launcher. That feels too broad for behavior currently motivated by specific Newton camera tasks.

In particular, I don’t think DirectRLEnv should auto-adjust observation_space from camera config fields. The task owns the observation layout, so it should explicitly stack frames and declare the resulting observation shape. If this becomes common across tasks, we can promote a shared helper later with clearer boundaries.

For example, for CartpoleCameraEnv, I’d rather see something like a task config field:

camera_frame_stack: int = 1
observation_space = [tiled_camera.height, tiled_camera.width, 3 * camera_frame_stack]

and then handle the history directly in CartpoleCameraEnv._get_observations():

def _stack_camera_obs(self, camera_data: torch.Tensor) -> torch.Tensor:
    frame_stack = self.cfg.camera_frame_stack
    if frame_stack <= 1:
        return camera_data
    if self._camera_frame_history is None:
        self._camera_frame_history = camera_data.unsqueeze(0).repeat(frame_stack, 1, 1, 1, 1)
        self._camera_frame_stack_idx = 0
    self._camera_frame_history[self._camera_frame_stack_idx].copy_(camera_data)
    stacked = torch.cat(
        [
            self._camera_frame_history[(self._camera_frame_stack_idx + 1 + i) % frame_stack]
            for i in range(frame_stack)
        ],
        dim=-1,
    )
    self._camera_frame_stack_idx = (self._camera_frame_stack_idx + 1) % frame_stack
    return stacked

Then _get_observations() can remain explicit:

camera_data = self._tiled_camera.data.output[data_type] / 255.0
camera_data -= torch.mean(camera_data, dim=(1, 2), keepdim=True)
camera_data = self._stack_camera_obs(camera_data)
return {"policy": camera_data.clone()}

Reset handling can also stay task-local by clearing or reinitializing the history for env_ids in _reset_idx(). That keeps the observation contract, memory cost, reset behavior, and Newton-specific policy choice visible in the environment that actually needs it.

I’d also document the Newton/Warp renderer class subtlety directly: unlike RTX modes with temporal AA, the Warp renderer does not inject prior-frame information into the current image. Camera-control tasks that need velocity-like visual cues should therefore add explicit temporal observations, such as task-local frame stacking, rather than relying on renderer-specific artifacts.

Feel free to give my comment to agent and and have it work on next iteration x)

[jmart-nv]

Closing this PR in favor of per-task frame stacking