Enhances Pink IK controller with null-space posture control and improv…#3149
Merged
Conversation
michaellin6
requested review from
huihuaNvidia2023,
pascal-roth,
peterd-NV and
rwiltz
michaellin6
requested review from
Mayankm96,
jtigue-bdai and
kellyguo11
as code owners
Contributor
|
Nice work. Overall it looks good to me. |
Collaborator
|
Do we have numbers to validate Mimic datagen/Robomimic policy success rate on the existing teleoperation/imitation learning workflows? Demo 1 and Demo 2 here. @michaellin6 |
6 tasks
Contributor
Author
I am working on getting these metrics |
michaellin6
force-pushed
the
mlin/teleop-ik-improv
branch
2 times, most recently
from
00f3c19 to
1bd63bf
Compare
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
requested changes
Aug 16, 2025
Contributor
Mayankm96
left a comment
Mayankm96
left a comment
There was a problem hiding this comment.
I did a high-level pass of the code. Most of it seems fine. Since I'm not familiar with PINK and mimic related tasks, I don't have any comments on the implementation side. It would be great if someone from the Mimic team also goes over it.
From high-level, it seems there are some mimic tasks in isaaclab_mimic and some in isaaclab_tasks. It would be great to keep them all together as much as possible. We can discuss this offline as it's more about restructuring the code to make it easier to navigate through it.
Mayankm96
reviewed
Aug 16, 2025
Mayankm96
reviewed
Aug 16, 2025
Contributor
huihuaNvidia2023
left a comment
huihuaNvidia2023
left a comment
There was a problem hiding this comment.
First pass done. Will take more time on the null space posture task.
Contributor
Author
@peterd-NV Values are updated in the PR description. We will need to update the datasets provided. |
michaellin6
force-pushed
the
mlin/teleop-ik-improv
branch
from
2affaeb to
e3940bc
Compare
ooctipus
reviewed
Aug 19, 2025
ooctipus
reviewed
Aug 19, 2025
ooctipus
reviewed
Aug 19, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
reviewed
Aug 26, 2025
Mayankm96
approved these changes
Aug 26, 2025
Contributor
Mayankm96
left a comment
Mayankm96
left a comment
There was a problem hiding this comment.
Mostly NITs. Would require to run pre-commit once addressed.
Approving otherwise.
Contributor
|
[like] Huihua Zhao reacted to your message:
…________________________________
From: Mayank Mittal ***@***.***>
Sent: Tuesday, August 26, 2025 4:30:17 PM
To: isaac-sim/IsaacLab ***@***.***>
Cc: Huihua Zhao ***@***.***>; Mention ***@***.***>
Subject: Re: [isaac-sim/IsaacLab] Enhances Pink IK controller with null-space posture control and improv… (PR #3149)
@Mayankm96 approved this pull request.
Mostly NITs. Would require to run pre-commit once addressed.
Approving otherwise.
—
Reply to this email directly, view it on GitHub<#3149 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHYFEGJZVW5QRC6JIBI7UB33PSDRTAVCNFSM6AAAAACDUUZQXCVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCNJWGQ2DONRQGE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
peterd-NV
reviewed
Aug 26, 2025
peterd-NV
approved these changes
Aug 26, 2025
ooctipus
approved these changes
Aug 26, 2025
Contributor
|
[celebrate] Huihua Zhao reacted to your message:
…________________________________
From: Kelly Guo ***@***.***>
Sent: Wednesday, August 27, 2025 3:09:56 AM
To: isaac-sim/IsaacLab ***@***.***>
Cc: Huihua Zhao ***@***.***>; Mention ***@***.***>
Subject: Re: [isaac-sim/IsaacLab] Enhances Pink IK controller with null-space posture control and improv… (PR #3149)
Merged #3149<#3149> into main.
—
Reply to this email directly, view it on GitHub<#3149 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHYFEGNQPJIZE77XP2ZTRVL3PUOQJAVCNFSM6AAAAACDUUZQXCVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJZGM3DAMJSGA2TGMY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
This was referenced Aug 27, 2025
kellyguo11
pushed a commit
that referenced
this pull request
Aug 28, 2025
…port error (#3292) # Description Feature introduced in #3149 imports pink_ik for isaaclab.controller module. This is causing an error IsaacLab wide due to pinocchio import. This PR removes the import. Fixes # (issue) Fixed pinocchio import error due to isaaclab.controller module importing pink_ik by default. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Breaking change (fix or feature that would cause existing functionality to not work as expected) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
kellyguo11
added a commit
that referenced
this pull request
Sep 9, 2025
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia These changes introduce new environments: `Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and `Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments, specially the locomanipulation one, feature: 1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro 2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet. 3. Integration with upper body controller that is driven by Inverse Kinematics. Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well. <img width="645" height="275" alt="image" src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2" /> This code has been tested through the [Mimic pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html), and the output dataset was used to [train a robomimic policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent) that achieves a success rate of **92% from 50 policy rollouts**. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> Some fixes included in these changes: 1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter. 2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration `class:PinkKinematicsConfiguration` that allows for all joints of an articulation to be updated while only a subset are controlled by IK. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshots  Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there⚠️ This PR depends on #3149 and should only be merged after that PR <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
george-nehma
pushed a commit
to george-nehma/DreamLander-IsaacLab
that referenced
this pull request
Oct 24, 2025
…v… (isaac-sim#3149) # Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Enhance Pink IK Controller with Null Space Posture Control This PR improves the Pink IK controller integration for better humanoid robot control and more natural postures. **Note**: Original this PR was staged in the internal repo (isaac-sim#547). It has been moved here due to new Github workflow. ## Key Changes ### New Null Space Posture Task - Added NullSpacePostureTask to enforce postural constraints on shoulder/waist joints while prioritizing end-effector tasks - Maintains natural robot poses during manipulation ### Controller Improvements - Tuned low level PD controller gains - Support mixed task types (FrameTask + NullSpacePostureTask) ### Testing & Environment Updates - Redesigned pink controller test script to use JSON-based configurations to program test motions. - Updated all environments (PickPlace, NutPour, ExhaustPipe) with null space control, damping tasks, and improved tracking - Added `Isaac-PickPlace-GR1T2-WaistEnabled-Abs-v0` env that is identical to `Isaac-PickPlace-GR1T2-Abs-v0` but enables the Waist DOFs. - Added target_eef_link_names mapping for clearer link specification Fixes # (issue) These changes help fix the following problems from [VDR feedback](https://docs.google.com/document/d/1saB1QA5r_WlD1l17q7C04WWNltnBW-K0ydI2UB8jxAs/edit?tab=t.0) - [Enable Waist DOF](https://nvbugspro.nvidia.com/bug/5235527) - Discourage elbow flare - Make controller low-latency and low-jerk. **We improved the unit test for the pink controller and reduced our position and rotation accuracy tolerance from 30 mm, 10 degrees to 1 mm, 1 degree.** - Develop metric for controller performance - Added a flag to disable failure due to joint limits. Previously, any commanded pose that ended in joint limit violation would result in no solution and the controlled robot freezing in place. This change gets the solver to still provide a solution and instead issue a warning for joint limit violations. ## Screenshots These controller changes have been tested through the Mimic pipeline (teleop_se3_agent.py, record/replay_demos.py). Here are videos showing teleoperation of all three environments working. ### PickPlace-GR1T2-Abs  ### NutPour-GR1T2  ### ExhaustPipe-GR1T2  ### Successfully Trained Robomimic Model Rollout on PickPlace task For the two robomimic tasks: `Isaac-PickPlace-GR1T2-Abs-v0` and `Isaac-NutPour-GR1T2-Pink-IK-Abs-v0`, if we collect a new dataset, we achieve a success rate of 96 and 92% respectively.   ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
george-nehma
pushed a commit
to george-nehma/DreamLander-IsaacLab
that referenced
this pull request
Oct 24, 2025
…port error (isaac-sim#3292) # Description Feature introduced in isaac-sim#3149 imports pink_ik for isaaclab.controller module. This is causing an error IsaacLab wide due to pinocchio import. This PR removes the import. Fixes # (issue) Fixed pinocchio import error due to isaaclab.controller module importing pink_ik by default. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Breaking change (fix or feature that would cause existing functionality to not work as expected) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
hougantc-nvda
pushed a commit
to hougantc-nvda/IsaacLab
that referenced
this pull request
Nov 10, 2025
…im#3150) <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia These changes introduce new environments: `Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and `Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments, specially the locomanipulation one, feature: 1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro 2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet. 3. Integration with upper body controller that is driven by Inverse Kinematics. Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well. <img width="645" height="275" alt="image" src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2" /> This code has been tested through the [Mimic pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html), and the output dataset was used to [train a robomimic policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent) that achieves a success rate of **92% from 50 policy rollouts**. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> Some fixes included in these changes: 1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter. 2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration `class:PinkKinematicsConfiguration` that allows for all joints of an articulation to be updated while only a subset are controlled by IK. <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality)  Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there⚠️ This PR depends on isaac-sim#3149 and should only be merged after that PR <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
hougantc-nvda
pushed a commit
to hougantc-nvda/IsaacLab
that referenced
this pull request
Nov 10, 2025
…im#3150) <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia These changes introduce new environments: `Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and `Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments, specially the locomanipulation one, feature: 1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro 2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet. 3. Integration with upper body controller that is driven by Inverse Kinematics. Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well. <img width="645" height="275" alt="image" src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2" /> This code has been tested through the [Mimic pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html), and the output dataset was used to [train a robomimic policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent) that achieves a success rate of **92% from 50 policy rollouts**. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> Some fixes included in these changes: 1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter. 2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration `class:PinkKinematicsConfiguration` that allows for all joints of an articulation to be updated while only a subset are controlled by IK. <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality)  Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there⚠️ This PR depends on isaac-sim#3149 and should only be merged after that PR <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
4 tasks
ooctipus
pushed a commit
that referenced
this pull request
May 16, 2026
…_ik (#5644) ## Summary One-character fix in `source/isaaclab/test/controllers/test_pink_ik.py:309`: ```diff - quat_from_matrix(matrix_from_quat(target_rot_tensor) * matrix_from_quat(quat_inv(current_rot))) + quat_from_matrix(matrix_from_quat(target_rot_tensor) @ matrix_from_quat(quat_inv(current_rot))) ``` `calculate_rotation_error` was composing two rotation matrices with PyTorch's element-wise multiplication (`*`) where matrix multiplication (`@`) was intended. The Hadamard product of two rotation matrices is not generally a rotation matrix. ## Why this surfaced as test failures now The bug has been latent since [#3149](#3149) (2025-08-26) because the Hadamard product of two near-identity matrices is also near-identity — `quat_from_matrix` could still recover a near-unit quaternion and the assertion `rot_error ≈ 0` would pass for completely wrong mathematical reasons. It became visible when [#5609 (jmart)](#5609) (2026-05-14) added the unit-norm guard to `isaaclab/utils/math.py:quat_from_matrix`: ```python invalid = (quat.norm(p=2, dim=-1, keepdim=True) - 1.0).abs() > 2e-5 return torch.where(invalid, torch.full_like(quat, float("nan")), quat) ``` After that PR, any non-rotation input (the Hadamard mess) returns NaN, which `axis_angle_from_quat` propagates → `torch.max(NaN) = NaN` → `AssertionError: Left hand IK rotation error (nan) exceeds tolerance`. Both hands always went to NaN; left hand is just asserted first. ## Verification Local repro on the Horde VM against current `develop` (`isaaclab_physx` backend, `newton[sim]@v1.2.0rc2`): | Configuration | Result | |---|---| | Unfixed, `Isaac-PickPlace-GR1T2-Abs-v0-horizontal_movement` | FAILED — `Left hand IK rotation error (nan)` | | Fixed, same parameterization | PASSED — rotation errors `1e-4` to `1e-7` (well within 0.02 rad tolerance) | | Fixed, all 12 GR1T2 cases, run 1 | 11 passed, 1 skipped | | Fixed, all 12 GR1T2 cases, run 2 | 11 passed, 1 skipped (deterministic) | ## Scope This addresses the consistent `Left hand IK rotation error (nan)` failures seen across recent develop PRs (e.g. [#5633 `test-curobo` log](https://github.com/isaac-sim/IsaacLab/actions/runs/25926139790/job/76211194676), [#5609 `test-curobo` log](https://github.com/isaac-sim/IsaacLab/actions/runs/25831490295/job/75897258188), [#5616 `test-curobo` log](https://github.com/isaac-sim/IsaacLab/actions/runs/25930392313/job/76222556444)). Remaining failures on G1 envs (finite ~0.03-0.05 rad rotation errors against the 0.030 rad tolerance) are a **separate** issue — IK convergence quality rather than the NaN math bug. Out of scope for this PR; needs its own ticket. ## Test plan - [x] Pre-commit clean. - [x] Unfixed branch reproduces NaN on `Isaac-PickPlace-GR1T2-Abs-v0-horizontal_movement` locally. - [x] Fixed branch passes the same parameterization locally with finite rotation errors. - [x] Fixed branch passes all 12 GR1T2 parameterizations across two consecutive runs (deterministic).
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enhance Pink IK Controller with Null Space Posture Control
This PR improves the Pink IK controller integration for better humanoid robot control and more natural postures.
Note: Original this PR was staged in the internal repo (#547). It has been moved here due to new Github workflow.
Key Changes
New Null Space Posture Task
Controller Improvements
Testing & Environment Updates
Isaac-PickPlace-GR1T2-WaistEnabled-Abs-v0env that is identical toIsaac-PickPlace-GR1T2-Abs-v0but enables the Waist DOFs.Fixes # (issue)
These changes help fix the following problems from VDR feedback
Screenshots
These controller changes have been tested through the Mimic pipeline (teleop_se3_agent.py, record/replay_demos.py). Here are videos showing teleoperation of all three environments working.
PickPlace-GR1T2-Abs
NutPour-GR1T2
ExhaustPipe-GR1T2
Successfully Trained Robomimic Model Rollout on PickPlace task
For the two robomimic tasks:
Isaac-PickPlace-GR1T2-Abs-v0andIsaac-NutPour-GR1T2-Pink-IK-Abs-v0, if we collect a new dataset, we achieve a success rate of 96 and 92% respectively.Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there