ProxyArray and Asset/Sensor level property caching#5304
Merged
Conversation
Warp-first dual-access array wrapper for articulation data properties. Provides explicit .torch and .warp accessors with a deprecation bridge for smooth migration from wp.to_torch() patterns.
List concrete test suites and pre-commit checks to run during implementation.
Six tasks: TorchArray class (TDD), base type hints, PhysX data, Newton data, validation tests, changelogs.
Warp-first dual-access array wrapper. Provides cached zero-copy .torch view and .warp accessor for kernel interop. Includes a deprecation bridge (__torch_function__ + operators) for migration.
Cover __radd__, __sub__, __mul__, __neg__, and comparison operators individually to prevent regressions.
Change all property return type hints from wp.array to TorchArray. Wrap deprecated property returns in TorchArray().
Wrap all property returns in TorchArray(). Internal warp kernels access sibling properties via .warp for wp.launch() compatibility.
Same pattern as PhysX: wrap returns in TorchArray(), access sibling properties via .warp for wp.launch() compatibility.
Update three packages with new version entries documenting the TorchArray feature that wraps wp.array with explicit .torch and .warp accessors: - isaaclab 4.6.1 -> 4.6.2 - isaaclab_physx 0.5.16 -> 0.5.17 - isaaclab_newton 0.5.13 -> 0.5.14
Internal code that passes data properties to wp.launch(), wp.zeros_like(), wp.clone(), and wp.to_torch() now uses .warp or .torch explicitly.
Internal code that passes data properties to wp.launch(), wp.zeros_like(), and wp.to_torch() now uses .warp or .torch explicitly.
Replace wp.to_torch(data.property) with data.property.torch throughout the test file for data properties that now return TorchArray. Calls on root_view methods and wrench_composer properties are kept as wp.to_torch() since those still return plain warp arrays.
Replace wp.to_torch(data.property) with data.property.torch in MDP termination functions. Add .warp in wrench composer lambda.
Replace wp.to_torch(data.property) with data.property.torch throughout the test file.
Replace wp.to_torch(data.property) with data.property.torch across observations, rewards, events, actions, and commands.
Replace wp.to_torch(data.property) with data.property.torch across observations, rewards, events, actions, commands, and cartpole task.
Replace wp.to_torch(data.property) with data.property.torch.
Replace wp.to_torch(data.property) with data.property.torch across all manager-based RL task environments.
Replace wp.to_torch(data.property) with data.property.torch across all direct RL task environments.
Update base and PhysX PVA/IMU data classes to return TorchArray. Update tests to use .torch accessor instead of wp.to_torch().
Update PhysX DeformableObject data class to return TorchArray from all properties and class-level attributes. Fix asset class consumers to use .warp for kernel inputs and .torch for torch contexts. Update tests, tutorial scripts, and interactive_scene to use .torch instead of wp.to_torch().
Update base, PhysX, and Newton FrameTransformer data classes to return TorchArray. Fix consumers and update tests.
Update base, PhysX, and Newton RigidObjectCollection data classes to return TorchArray. Fix asset class consumers and update tests.
Update base, PhysX, and Newton RigidObject data classes to return TorchArray. Fix asset class consumers and update tests.
Update base, PhysX, and Newton ContactSensor data classes to return TorchArray. Fix sensor class consumers and update tests.
GRAVITY_VEC_W is a raw wp.array constant, not a TorchArray property.
default_joint_pos/vel need .torch before passing to write_joint_state_to_sim.
Aligns BaseFrameView.get_world_poses / get_local_poses and the UsdFrameView / FabricFrameView / NewtonSiteFrameView backends with the sensor-data migration: the pose getters now return tuple[TorchArray, TorchArray] instead of tuple[wp.array, wp.array]. Callers use .torch for torch views and .warp for the underlying wp.array. Newton and Fabric wrap their persistent _pos_buf / _quat_buf (and Newton's local-pose siblings) in TorchArrays once at buffer allocation, returning the cached wrappers for the indices=None fast path. USD and all indexed paths construct a fresh TorchArray per call wrapping the fresh wp.array output. No rebind plumbing - backing buffers are never re-allocated in the current codebase. Breaking change for downstream code that wrapped these returns in wp.to_torch(...) or passed them directly to a warp kernel - see CHANGELOG for migration.
Bumps isaaclab 4.6.14 -> 4.6.15 (breaking change entry for the BaseFrameView pose-getter return type), and adapter entries for isaaclab_physx 0.5.22 -> 0.5.23 (FabricFrameView) and isaaclab_newton 0.5.22 -> 0.5.23 (NewtonSiteFrameView).
Updates all tutorial, demo, state-machine, and imitation-learning scripts under scripts/ so they access TorchArray data via .torch instead of wp.to_torch(sensor.data.X). Post-TorchArray-migration, wp.to_torch on a migrated property raises TypeError because TorchArray has no requires_grad attribute. Adds source/isaaclab/test/test_scripts_torcharray_patterns.py as a static scanner that walks scripts/ and fails if the old pattern is reintroduced. Covers both wp.to_torch(<chain>.data.<field>) and the aliased wp.to_torch(<name>_data.<field>) form. Skips scripts/tools/wrap_warp_to_torch.py which documents the old pattern in strings as part of an older migration utility.
…x-sim-bindings # Conflicts: # source/isaaclab_physx/config/extension.toml # source/isaaclab_physx/docs/CHANGELOG.rst
Hard rename of the warp-backed dual-view wrapper from TorchArray to ProxyArray across source, scripts, and docs. "ProxyArray" better describes what the class is: a proxy around a wp.array exposing both .warp and .torch accessors, rather than a torch-first type. - Class rename: TorchArray -> ProxyArray - Module rename: isaaclab.utils.warp.torch_array -> proxy_array - File renames: torch_array.py -> proxy_array.py, test_torch_array.py -> test_proxy_array.py, docs/source/how-to/torch_array.rst -> proxy_array.rst - No backward-compatibility alias.
Removes ProxyArray.rebind and overrides __setattr__ to raise on any post-init attribute assignment except the internal _torch_cache. The lazy torch cache still populates on first .torch access; everything else is frozen. All prior rebind call sites in Newton and PhysX asset data classes now construct a new ProxyArray when the underlying sim-bound buffer is re-created, instead of mutating an existing wrapper. The data class itself keeps its attribute mutability, so ``self._foo_ta = ProxyArray(new_buf)`` is the replacement idiom. Rationale: mutable rebind was unsafe — any caller holding the old torch view silently read stale data after a rebind. Forcing a new ProxyArray makes the stale-reference hazard explicit and local.
Documents the class rename TorchArray -> ProxyArray (module rename isaaclab.utils.warp.torch_array -> .proxy_array) and the removal of ProxyArray.rebind in favor of constructing a fresh wrapper when the underlying buffer is re-created.
Catches identifier-embedded occurrences that the initial word-boundary regex skipped: * Test helpers _check_torch_array -> _check_proxy_array (and ~250 call sites across test_rigid_object_iface.py, test_rigid_object_collection_iface.py, test_articulation_iface.py). * Data-class methods _pin_torch_arrays -> _pin_proxy_arrays across PhysX, OvPhysx, and Newton articulation/rigid-object data. * Test class names TestTorchArrayBasic and siblings -> TestProxyArray*. * Prose comments referring to "TorchArrays" / "sim-bound TorchArrays" updated to "ProxyArrays".
Two PhysX root_view call sites pass public data properties directly to omni.physics.tensors setters, which read .ptr on the argument. After the TorchArray -> ProxyArray migration those properties return ProxyArray, not wp.array, so the setters fail with ``AttributeError: 'ProxyArray' object has no attribute 'ptr'``. Extracts .warp at the call site for: * write_fixed_tendon_properties_to_sim_index (stiffness, damping, limit_stiffness, pos_limits, rest_length, offset) * write_spatial_tendon_properties_to_sim_index (stiffness, damping, limit_stiffness, offset) Also fixes test_articulation_data.py in isaaclab_ovphysx which called .numpy() directly on a ProxyArray; now uses .warp.numpy().
Third-party and legacy code that still uses ``wp.to_torch(asset.data.<field>)`` broke after the TorchArray -> ProxyArray migration because ProxyArray deliberately doesn't replicate the full ``wp.array`` attribute surface (e.g. ``requires_grad``), so warp's ``to_torch`` raised AttributeError. Installs a single monkey-patch at ``isaaclab.utils.warp`` import time. The shim: * Detects ProxyArray inputs and returns the cached ``.torch`` view (zero-copy, same underlying memory). * Emits a one-shot DeprecationWarning pointing callers to the new ``.torch`` accessor. * Forwards anything else to the original ``wp.to_torch``, preserving the ``requires_grad`` kwarg. Patches both ``wp.to_torch`` and ``wp._src.torch.to_torch`` so the rare ``from warp._src.torch import to_torch`` import style is also covered. The shim installs once — Python's attribute lookup on the ``warp`` module picks up the new reference for every subsequent ``wp.to_torch(...)`` call. Adds four regression tests in TestWpToTorchShim covering raw wp.array pass-through, ProxyArray routing, the one-shot warning, and the requires_grad kwarg.
kellyguo11
approved these changes
Apr 26, 2026
AntoineRichard
changed the title
…x-sim-bindings # Conflicts: # docs/source/migration/migrating_to_isaaclab_3-0.rst # source/isaaclab/docs/CHANGELOG.rst # source/isaaclab_newton/config/extension.toml # source/isaaclab_newton/docs/CHANGELOG.rst # source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation_data.py # source/isaaclab_newton/isaaclab_newton/assets/rigid_object/rigid_object_data.py # source/isaaclab_physx/config/extension.toml # source/isaaclab_physx/docs/CHANGELOG.rst # source/isaaclab_tasks/config/extension.toml # source/isaaclab_tasks/docs/CHANGELOG.rst
The Warp-graphable MDP terms and the Warp inhand-manipulation env were reading asset/sensor data properties (now ProxyArray) and passing the result straight to wp.launch. The implicit __cuda_array_interface__ bridge made this work, but is not explicit. Use ProxyArray.warp at the call sites where the value flows into a warp kernel (or a sim-write helper that forwards to one), and drop the redundant wp.to_torch round-trip in ObservationManager since ProxyArray forwards .shape directly. No behavior change.
Use .warp.shape instead of the implicit ProxyArray.shape forwarding in the experimental joint_pos_out_of_manual_limit termination and the ObservationManager._resolve_out_dim "joint" path. ProxyArray.shape already delegates to the warp shape, so behavior is unchanged — but spelling it .warp.shape signals intent and dodges the warp-vs-torch shape ambiguity for vector dtypes (e.g. transformf would expose 1D in warp but 2D in torch).
Adds a small ``proxy_array(*args, **kwargs)`` helper that wraps a ``wp.array(...)`` constructor in :class:`ProxyArray`, and refactors :class:`MockArticulationData` and :class:`MockContactSensorData` to use it. The mocks now read as ``self.joint_pos = proxy_array(np_data, device=...)`` instead of nesting ``ProxyArray(wp.array(...))`` at every site. Behavior unchanged. Also keeps the previously-added ``copy_np_to_wp`` ProxyArray-aware unwrap and the ``.warp.ndim`` probe in ``mutate_root_state`` so the in-place mutators continue to work on the wrapped attributes.
After the merge, asset/sensor data class properties return ProxyArray.
``wp.to_torch(<ProxyArray>)`` rides the __cuda_array_interface__
deprecation bridge: it works but emits a DeprecationWarning and
allocates a fresh torch tensor every call instead of reusing the
ProxyArray's cached zero-copy view.
Replace with the explicit ``.torch`` accessor at the source bug in
the velocity terrain curriculum and at the test sites that read
art_data.{joint_pos,joint_vel,default_joint_pos,default_joint_vel,
soft_joint_pos_limits,soft_joint_vel_limits} in
test_events_warp_parity and test_actions_warp_parity. Drop the
now-unused ``import warp as wp`` from curriculums.py.
Same Warp-graphable migration as the previous experimental commit, now applied to the isaaclab_tasks_experimental MDPs (humanoid, cartpole, locomotion, reach) and the warp direct envs (cartpole_warp_env, locomotion_env_warp). Asset/sensor data property reads that flow into ``wp.launch(inputs=[...])`` (or into a sim-write helper that forwards to one) now use the explicit ``.warp`` accessor instead of relying on ProxyArray's ``__cuda_array_interface__`` deprecation bridge. Side-effects: - ``cartpole_warp_env`` and ``locomotion_env_warp`` cache their simulation bindings as ``self.X = self.robot.data.X.warp`` so callers get a stable wp.array reference. - ``cartpole/mdp/rewards.joint_pos_target_l2`` reads ``.warp.shape[1]`` for the joint-mask sanity check (warp shape is the right one for scalar dtypes; explicit form avoids ambiguity). No behavior change.
The Newton asset data classes have many internal lazy-derived properties (root_link_vel_w, root_com_pose_w, projected_gravity_b, heading_w, body-frame velocities, deprecated *_state_w, ...) that launch warp kernels on demand. Their inputs include public ProxyArray properties such as root_link_pose_w, root_com_vel_w, body_com_pos_b, GRAVITY_VEC_W, FORWARD_VEC_B, root_link_quat_w, etc. These were passed directly to wp.launch and rode ProxyArray's __cuda_array_interface__ deprecation bridge. Switch every such read to the explicit .warp accessor in: - articulation_data.py - rigid_object_data.py - rigid_object_collection_data.py Private TimestampedBuffer outputs (self._foo.data) and sim-bind/previous-snapshot wp.array attributes (self._sim_bind_*, self._previous_*) are left alone -- they are already raw wp.array. No behavior change.
ProxyArray now reads the WARN_ON_TORCH_QUATF_ACCESS environment variable on every .torch access. When set to "1", any read of a wp.quatf-typed ProxyArray emits a UserWarning that points at the caller's source line (stacklevel=2) and explains that the Isaac Lab quaternion convention changed from (w, x, y, z) in 2.x to (x, y, z, w) in 3.x. The detector is silent by default; users opt in during migration with `export WARN_ON_TORCH_QUATF_ACCESS=1`, walk through the warning sites, then unset for production runs. The dtype check is cached on construction so the .torch read path stays a constant-time check; only ProxyArrays wrapping wp.quatf arrays even consult the env var. Documents the new tool in the Isaac Lab 3.0 migration guide as a runtime complement to the source-level Quaternion Finder Tool, and adds four unit tests (env unset, env=1 on quatf, env=1 on non-quatf, env=0 on quatf) under test_proxy_array.
This was referenced Apr 27, 2026
mmichelis
pushed a commit
to mmichelis/IsaacLab
that referenced
this pull request
Apr 29, 2026
# Description Follows the TorchArray concept from MJLab and augment it with some lab requirements. ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [ ] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
AntoineRichard
added a commit
that referenced
this pull request
May 8, 2026
## Summary - Migrates core test and MDP callers off deprecated state/read/write helper APIs. - Updates the test_pose_inv tensor-to-NumPy conversion for NumPy 2.0. - Bumps the isaaclab changelog/version because core MDP source changed. ## Verification - ./isaaclab.sh -f - Scoped deprecated-call-site search: assigned core matches removed. Rebased onto develop after PR #5304 merged.
AntoineRichard
added a commit
that referenced
this pull request
May 8, 2026
## Summary - Migrates task/contrib camera callers from TiledCamera aliases to Camera. - Updates task state reads and in-hand write/target helper calls to explicit APIs. - Bumps task/contrib changelogs and extension versions for touched packages. ## Verification - ./isaaclab.sh -f - Scoped deprecated-call-site search: concrete task/contrib deprecated calls removed. Rebased onto develop after PR #5304 merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follows the TorchArray concept from MJLab and augment it with some lab requirements.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there