Skip to content

Fix unspecified device in Newton Renderer camera transforms#5019

Merged
ooctipus merged 3 commits into
isaac-sim:developfrom
nblauch:develop
Mar 14, 2026
Merged

Fix unspecified device in Newton Renderer camera transforms#5019
ooctipus merged 3 commits into
isaac-sim:developfrom
nblauch:develop

Conversation

@nblauch
Copy link
Copy Markdown

@nblauch nblauch commented Mar 14, 2026

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Mar 14, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 14, 2026

Greptile Summary

This PR fixes a bug in RenderData.update() where wp.empty() and wp.launch() for the camera_transforms array were missing an explicit device argument, defaulting to Warp's default device instead of render_context.device. This could cause device-mismatch runtime errors when the simulation runs on a non-default device.

  • Fix is correct: device=self.render_context.device is now properly passed to both wp.empty() (array allocation) and wp.launch() (kernel execution), ensuring the array lives on and the kernel runs on the renderer's target device.
  • Dtype inconsistency introduced: wp.empty() now uses dtype=wp.transform, while the class-level type annotation (line 60), kernel parameter (line 140), and kernel body (line 143) all continue to use wp.transformf. These are equivalent types in Warp, so there is no functional issue, but the inconsistency is worth cleaning up.
  • CHANGELOG not updated: The PR checklist shows the changelog entry as unchecked; the fix should be recorded in source/isaaclab_newton/docs/CHANGELOG.rst under the current version.
  • Input tensor device: The positions and converted_orientations PyTorch tensors are passed to wp.launch without explicit verification that they reside on render_context.device. This is pre-existing behaviour, but is worth keeping in mind if multi-GPU scenarios are supported in the future.

Confidence Score: 4/5

  • This PR is safe to merge; it correctly fixes a real device-mismatch bug with a targeted, minimal change
  • The fix is clearly correct — missing device arguments on wp.empty and wp.launch would cause allocation and kernel execution on the wrong device. The only concern is a minor wp.transform / wp.transformf naming inconsistency introduced in the same hunk, which is functionally harmless since the two types are aliases in Warp. No tests are added, and the CHANGELOG was not updated.
  • No files require special attention beyond the single changed file.

Important Files Changed

Filename Overview
source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py Correctly adds device=self.render_context.device to wp.empty() and wp.launch() in RenderData.update(), fixing device-unspecified allocations that could cause device mismatch errors; minor inconsistency between wp.transform (new code) and wp.transformf (class annotation, kernel signature, kernel body)

Sequence Diagram

sequenceDiagram
    participant Sensor
    participant RenderData
    participant WarpRuntime

    Sensor->>RenderData: update(positions, orientations, intrinsics)
    RenderData->>WarpRuntime: wp.empty((1, world_count), dtype=wp.transform,<br/>device=render_context.device)
    WarpRuntime-->>RenderData: camera_transforms [on render_context.device]
    RenderData->>WarpRuntime: wp.launch(_update_transforms, world_count,<br/>[positions, orientations, camera_transforms],<br/>device=render_context.device)
    WarpRuntime-->>RenderData: camera_transforms populated
    RenderData->>WarpRuntime: compute_pinhole_camera_rays(width, height, fov)
    WarpRuntime-->>RenderData: camera_rays
Loading

Last reviewed commit: bd52495

Comment thread source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py Outdated
nblauch and others added 2 commits March 13, 2026 21:51
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: Nicholas Blauch <nblauch@gmail.com>
@ooctipus ooctipus merged commit 7ac32c9 into isaac-sim:develop Mar 14, 2026
15 of 19 checks passed
daniela-hase pushed a commit to daniela-hase/IsaacLab that referenced this pull request Mar 30, 2026
…m#5019)

# Description

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->

Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (existing functionality will not work without user
modification)
- Documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Signed-off-by: Nicholas Blauch <nblauch@gmail.com>
Co-authored-by: nblauch <nblauch@nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
hujc7 pushed a commit to hujc7/IsaacLab that referenced this pull request Mar 30, 2026
…m#5019)

# Description

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->

Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (existing functionality will not work without user
modification)
- Documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Signed-off-by: Nicholas Blauch <nblauch@gmail.com>
Co-authored-by: nblauch <nblauch@nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
kellyguo11 added a commit that referenced this pull request May 15, 2026
## Description

Fixes `OVRTXRenderer` crash on multi-GPU systems when `sim.device` is
not `cuda:0`.

**Root cause:** A hardcoded `DEVICE = "cuda:0"` constant in
`ovrtx_renderer_kernels.py` was imported and used for all Warp kernel
launches and buffer allocations. Additionally, `AttributeBinding.map()`
calls used `device_id=0`, pinning attribute mapping to GPU 0 regardless
of the simulation device.

**Fix:** 
- Remove the `DEVICE` constant and use `self._device` (set from
`CameraRenderSpec.device`) for all Warp operations (11 locations)
- Add `_device_id` property to extract the CUDA device index from the
device string
- Pass `device_id=self._device_id` to `AttributeBinding.map()` calls (2
locations: object binding and camera binding)

**Note on `RenderVarOutput.map()` calls:** These remain unchanged
(`device=Device.CUDA` only) because the OVRTX C API for render output
mapping (`ovrtx_map_output_description_t`) does not accept a `device_id`
parameter — the output is inherently mapped on whichever GPU OVRTX
rendered on.

**Total:** 13 hardcoded GPU-0 references fixed (11 Warp + 2
AttributeBinding).

This is the same bug class fixed for `NewtonRenderer` in #5019 — OVRTX
was not updated at that time.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and added my name to the
[`CONTRIBUTORS.md`](https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md)
or my organization to the
[`CONTRIBUTORS.md`](https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md)
list

---------

Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants