Adds Locomanipulation Environment with G1 for Mimic workflow#3150
Conversation
d2e5558 to
6951dbf
Compare
rwiltz
left a comment
There was a problem hiding this comment.
LGTM from the teleop side of things.
huihuaNvidia2023
left a comment
There was a problem hiding this comment.
Done the first pass. Will look more into the tests.
kellyguo11
left a comment
There was a problem hiding this comment.
can we upload agile_locomotion.pt to nucleus instead of hosting it in github?
6951dbf to
94e6cc1
Compare
|
Just rebased this branch to ToT. Will address all the previous review comments shortly. |
54cdf26 to
15ee2d7
Compare
huihuaNvidia2023
left a comment
There was a problem hiding this comment.
LTGM with two minor comments. Happy to discuss alternatives.
|
Hi, will the agile_locomotion training code be published in any form? |
4f82310 to
2ba1a9e
Compare
ooctipus
left a comment
There was a problem hiding this comment.
Implementation looks good to me,
Most concern were legacy issues regarding where to put robot - tasks specific files and specification on gym module registration. It might not be scope of this PR, but will be good to start address them earily
188e3d7 to
b605c6f
Compare
a87b280 to
ebbea14
Compare
|
Seeing timeout in |
20a558d to
f8b0a43
Compare
68b387b to
5bd7cba
Compare
… support - Implement fixed-base upper body IK and locomanipulation environments - Add G1 teleop with retargeter, hand rotation fixes, and comprehensive tests - Refactor locomotion to use retargeter instead of command manager - Enhance pink IK with kinematics model and LocalFrameTask for relative poses - Add locomanipulation policies with gravity compensation and tuned gains - Implement lower body standing retargeter with zero root velocity Co-authored-by: Michael Lin <michalin@nvidia.com> Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com>
… import error. Also increasing pink_ik test timeout
…m#3392) # Description Recent isaac sim update introduced a new bug for non-headless scripts where some scripts were hanging at simulation startup. This change introduces a new unit test that aims to capture issues like this by forcing the use of the non-headless app file. Additionally, the isaac sim CI system has very unstable results for perf testing, so we are disabling the performance-related tests for the sim CI. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
…ac-sim#3405) # Description The default_inertia attributes of the Articulation, RigidObjectCollection, and RigidObject data asset classes did not specify in what coordinate frame the tensors should be provided. This PR addresses this, and addresses some minor inconsistencies across the default_inertia docstrings. ## Type of change - This change requires a documentation update ## Screenshots ArticulationData | Before | After | | ------ | ----- | | <img width="731" height="162" alt="image" src="https://github.com/user-attachments/assets/7e2a6973-6264-4ba0-89d7-fd903e3bee6d" />| <img width="750" height="262" alt="image" src="https://github.com/user-attachments/assets/470832e6-0164-4402-b4c7-6a5ac5e599a1" />| RigidObjectCollectionData | Before | After | | ------ | ----- | | <img width="731" height="181" alt="image" src="https://github.com/user-attachments/assets/7d6bd039-b7f5-40e2-9180-33d0748694a8" /> | <img width="733" height="276" alt="image" src="https://github.com/user-attachments/assets/07bb32ca-628c-4132-8009-1db1c279d653" /> | RigidObjectData | Before | After | | ------ | ----- | | <img width="732" height="160" alt="image" src="https://github.com/user-attachments/assets/43b29f1c-3e92-4b96-925f-88cdf7ef3fc3" /> | <img width="733" height="276" alt="image" src="https://github.com/user-attachments/assets/d16bbc57-3939-4f2e-84cb-9ec135dcbe7e" /> | ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
048daf0 to
a9afae2
Compare
…im#3150) <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia These changes introduce new environments: `Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and `Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments, specially the locomanipulation one, feature: 1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro 2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet. 3. Integration with upper body controller that is driven by Inverse Kinematics. Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well. <img width="645" height="275" alt="image" src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2" /> This code has been tested through the [Mimic pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html), and the output dataset was used to [train a robomimic policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent) that achieves a success rate of **92% from 50 policy rollouts**. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> Some fixes included in these changes: 1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter. 2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration `class:PinkKinematicsConfiguration` that allows for all joints of an articulation to be updated while only a subset are controlled by IK. <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality)  Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there⚠️ This PR depends on isaac-sim#3149 and should only be merged after that PR <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
…im#3150) <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia These changes introduce new environments: `Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and `Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments, specially the locomanipulation one, feature: 1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro 2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet. 3. Integration with upper body controller that is driven by Inverse Kinematics. Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well. <img width="645" height="275" alt="image" src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2" /> This code has been tested through the [Mimic pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html), and the output dataset was used to [train a robomimic policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent) that achieves a success rate of **92% from 50 policy rollouts**. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> Some fixes included in these changes: 1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter. 2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration `class:PinkKinematicsConfiguration` that allows for all joints of an articulation to be updated while only a subset are controlled by IK. <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality)  Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there⚠️ This PR depends on isaac-sim#3149 and should only be merged after that PR <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
Description
Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia
These changes introduce new environments:
Isaac-PickPlace-Locomanipulation-G1-Abs-v0andIsaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0. These environments, specially the locomanipulation one, feature:Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well.

This code has been tested through the Mimic pipeline, and the output dataset was used to train a robomimic policy that achieves a success rate of 92% from 50 policy rollouts.
Fixes # (issue)
Some fixes included in these changes:
class:PinkKinematicsConfigurationthat allows for all joints of an articulation to be updated while only a subset are controlled by IK.Type of change
Screenshots
Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there