Skip to content

Adds Locomanipulation Environment with G1 for Mimic workflow#3150

Merged
kellyguo11 merged 12 commits into
isaac-sim:release/2.3.0from
michaellin6:team/locomanipulation-integration
Sep 9, 2025
Merged

Adds Locomanipulation Environment with G1 for Mimic workflow#3150
kellyguo11 merged 12 commits into
isaac-sim:release/2.3.0from
michaellin6:team/locomanipulation-integration

Conversation

@michaellin6
Copy link
Copy Markdown
Contributor

@michaellin6 michaellin6 commented Aug 12, 2025

Description

Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia

These changes introduce new environments: Isaac-PickPlace-Locomanipulation-G1-Abs-v0 and Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0. These environments, specially the locomanipulation one, feature:

  1. The G1 robot with the retargeter for 3-finger dexterous hand to operator hands tracked by Apple Vision Pro
  2. A locomotion policy trained with RL that enables G1 to walk and follow a input velocity command (vx, vy, wz, h) linear velocity in two axes, angular velocity around its pelvis' z-axis, and height distance of the pelvis from the feet.
  3. Integration with upper body controller that is driven by Inverse Kinematics.

Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well.
image

This code has been tested through the Mimic pipeline, and the output dataset was used to train a robomimic policy that achieves a success rate of 92% from 50 policy rollouts.

Fixes # (issue)

Some fixes included in these changes:

  1. Previously, Inverse kinematics relied on an input URDF of the robot to load a pinocchio kinematics model. This URDF was exported from the robot asset USD through USD to URDF export extension, however, if the extension is not setup properly, the joint ordering can change causing unexpected behavior due to mismatch of joint ordering between the IK controller and that of Isaac Lab. For the G1 asset, we are for now using a URDF asset directly instead of relying on the URDF exporter.
  2. Previously, the Inverse Kinematics controller assumed that for its kinematics model, only the joints that are being controlled by IK need to be updated, but when combined with other controllers on other joints this assumption breaks. Added a new kinematics configuration class:PinkKinematicsConfiguration that allows for all joints of an articulation to be updated while only a subset are controlled by IK.

Type of change

  • New feature (non-breaking change which adds functionality)

Screenshots

Locomanipulation Teacher Policy - Robomimic Policy
Robomimic policy for PickPlace trained with the Locomanipulation environment. We achieve 92% success rate from 50 rollouts.

Checklist

  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

⚠️ This PR depends on #3149 and should only be merged after that PR

@michaellin6 michaellin6 changed the title Implement Locomanipulation with Improved IK Control, G1 teleoperation and Locomotion policy Locomanipulation with Improved IK Control, G1 teleoperation and Locomotion policy Aug 12, 2025
@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch 2 times, most recently from d2e5558 to 6951dbf Compare August 14, 2025 22:52
Copy link
Copy Markdown
Contributor

@rwiltz rwiltz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from the teleop side of things.

Comment thread scripts/tools/replay_demos.py Outdated
Comment thread source/isaaclab_assets/isaaclab_assets/robots/unitree.py Outdated
Copy link
Copy Markdown
Contributor

@huihuaNvidia2023 huihuaNvidia2023 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done the first pass. Will look more into the tests.

Comment thread source/isaaclab/test/controllers/test_ik_configs/README.md Outdated
Comment thread source/isaaclab/isaaclab/controllers/pink_ik_cfg.py Outdated
@ooctipus ooctipus changed the title Locomanipulation with Improved IK Control, G1 teleoperation and Locomotion policy Improves IK Control for G1 teleoperation and Locomotion policy Aug 21, 2025
@michaellin6 michaellin6 changed the title Improves IK Control for G1 teleoperation and Locomotion policy Introducing Locomanipulation Environment with G1 for Mimic workflow Aug 24, 2025
Copy link
Copy Markdown
Contributor

@kellyguo11 kellyguo11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we upload agile_locomotion.pt to nucleus instead of hosting it in github?

Comment thread .gitattributes Outdated
@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from 6951dbf to 94e6cc1 Compare August 29, 2025 16:38
@michaellin6 michaellin6 requested a review from ooctipus as a code owner August 29, 2025 16:38
@michaellin6
Copy link
Copy Markdown
Contributor Author

Just rebased this branch to ToT. Will address all the previous review comments shortly.

Comment thread source/isaaclab/isaaclab/controllers/pink_ik/pink_ik_cfg.py Outdated
Comment thread source/isaaclab/test/controllers/test_pink_ik.py Outdated
Copy link
Copy Markdown
Contributor

@huihuaNvidia2023 huihuaNvidia2023 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM with two minor comments. Happy to discuss alternatives.

Comment thread source/isaaclab/isaaclab/controllers/pink_ik/pink_ik.py
@Cdfghglz
Copy link
Copy Markdown

Cdfghglz commented Sep 3, 2025

Hi, will the agile_locomotion training code be published in any form?

@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch 2 times, most recently from 4f82310 to 2ba1a9e Compare September 3, 2025 15:06
Copy link
Copy Markdown
Collaborator

@ooctipus ooctipus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation looks good to me,
Most concern were legacy issues regarding where to put robot - tasks specific files and specification on gym module registration. It might not be scope of this PR, but will be good to start address them earily

@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from 188e3d7 to b605c6f Compare September 5, 2025 05:49
@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from a87b280 to ebbea14 Compare September 6, 2025 00:47
Comment thread source/isaaclab/docs/CHANGELOG.rst
@kellyguo11
Copy link
Copy Markdown
Contributor

Seeing timeout in test_environments.py and failure in test_environments_with_stage_in_memory.py in CI. could you check locally if the tests are failing or whether we need to increase the timeouts for them?

@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from 20a558d to f8b0a43 Compare September 8, 2025 17:47
@michaellin6 michaellin6 removed the request for review from hhansen-bdai September 9, 2025 02:24
@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from 68b387b to 5bd7cba Compare September 9, 2025 02:30
michaellin6 and others added 10 commits September 8, 2025 21:57
… support

- Implement fixed-base upper body IK and locomanipulation environments
- Add G1 teleop with retargeter, hand rotation fixes, and comprehensive tests
- Refactor locomotion to use retargeter instead of command manager
- Enhance pink IK with kinematics model and LocalFrameTask for relative poses
- Add locomanipulation policies with gravity compensation and tuned gains
- Implement lower body standing retargeter with zero root velocity

Co-authored-by: Michael Lin <michalin@nvidia.com>
Co-authored-by: Huihua Zhao <huihuaz@nvidia.com>
Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com>
Co-authored-by: Sergey Grizan <sgrizan@nvidia.com>
… import error. Also increasing pink_ik test timeout
…m#3392)

# Description

Recent isaac sim update introduced a new bug for non-headless scripts
where some scripts were hanging at simulation startup. This change
introduces a new unit test that aims to capture issues like this by
forcing the use of the non-headless app file.

Additionally, the isaac sim CI system has very unstable results for perf
testing, so we are disabling the performance-related tests for the sim
CI.

## Type of change

- Bug fix (non-breaking change which fixes an issue)


## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x]  I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->
…ac-sim#3405)

# Description

The default_inertia attributes of the Articulation,
RigidObjectCollection, and RigidObject data asset classes did not
specify in what coordinate frame the tensors should be provided. This PR
addresses this, and addresses some minor inconsistencies across the
default_inertia docstrings.

## Type of change

- This change requires a documentation update

## Screenshots

ArticulationData

| Before | After |
| ------ | ----- |
| <img width="731" height="162" alt="image"
src="https://github.com/user-attachments/assets/7e2a6973-6264-4ba0-89d7-fd903e3bee6d"
/>| <img width="750" height="262" alt="image"
src="https://github.com/user-attachments/assets/470832e6-0164-4402-b4c7-6a5ac5e599a1"
/>|

RigidObjectCollectionData

| Before | After |
| ------ | ----- |
| <img width="731" height="181" alt="image"
src="https://github.com/user-attachments/assets/7d6bd039-b7f5-40e2-9180-33d0748694a8"
/> | <img width="733" height="276" alt="image"
src="https://github.com/user-attachments/assets/07bb32ca-628c-4132-8009-1db1c279d653"
/> |

RigidObjectData

| Before | After |
| ------ | ----- |
| <img width="732" height="160" alt="image"
src="https://github.com/user-attachments/assets/43b29f1c-3e92-4b96-925f-88cdf7ef3fc3"
/> | <img width="733" height="276" alt="image"
src="https://github.com/user-attachments/assets/d16bbc57-3939-4f2e-84cb-9ec135dcbe7e"
/> |


## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
@michaellin6 michaellin6 force-pushed the team/locomanipulation-integration branch from 048daf0 to a9afae2 Compare September 9, 2025 14:07
@michaellin6 michaellin6 changed the base branch from main to release/2.3.0 September 9, 2025 21:43
@kellyguo11 kellyguo11 merged commit ea5b6c6 into isaac-sim:release/2.3.0 Sep 9, 2025
7 of 8 checks passed
hougantc-nvda pushed a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 10, 2025
…im#3150)

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
-->
Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV,
@yanchangNvidia

These changes introduce new environments:
`Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and
`Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments,
specially the locomanipulation one, feature:
1. The G1 robot with the retargeter for 3-finger dexterous hand to
operator hands tracked by Apple Vision Pro
2. A locomotion policy trained with RL that enables G1 to walk and
follow a input velocity command (vx, vy, wz, h) linear velocity in two
axes, angular velocity around its pelvis' z-axis, and height distance of
the pelvis from the feet.
3. Integration with upper body controller that is driven by Inverse
Kinematics.

Below is a diagram showing the high level controller composition. The
environments introduced have the waist controlled by IK, but it is
possible to configure it such that waist DOFs are controlled by the
lower body policy as well.
<img width="645" height="275" alt="image"
src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2"
/>

This code has been tested through the [Mimic
pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html),
and the output dataset was used to [train a robomimic
policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent)
that achieves a success rate of **92% from 50 policy rollouts**.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->
Some fixes included in these changes:
1. Previously, Inverse kinematics relied on an input URDF of the robot
to load a pinocchio kinematics model. This URDF was exported from the
robot asset USD through USD to URDF export extension, however, if the
extension is not setup properly, the joint ordering can change causing
unexpected behavior due to mismatch of joint ordering between the IK
controller and that of Isaac Lab. For the G1 asset, we are for now using
a URDF asset directly instead of relying on the URDF exporter.
2. Previously, the Inverse Kinematics controller assumed that for its
kinematics model, only the joints that are being controlled by IK need
to be updated, but when combined with other controllers on other joints
this assumption breaks. Added a new kinematics configuration
`class:PinkKinematicsConfiguration` that allows for all joints of an
articulation to be updated while only a subset are controlled by IK.

<!-- As you go through the list, delete the ones that are not
applicable. -->

- New feature (non-breaking change which adds functionality)

![Locomanipulation Teacher Policy - Robomimic
Policy](https://github.com/user-attachments/assets/983b12af-f8e2-42bb-a942-92380348a843)
Robomimic policy for PickPlace trained with the Locomanipulation
environment. We achieve 92% success rate from 50 rollouts.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

⚠️ This PR depends on isaac-sim#3149  and should only be merged after that PR
<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Co-authored-by: Huihua Zhao <huihuaz@nvidia.com>
Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com>
Co-authored-by: Sergey Grizan <sgrizan@nvidia.com>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
hougantc-nvda pushed a commit to hougantc-nvda/IsaacLab that referenced this pull request Nov 10, 2025
…im#3150)

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
-->
Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV,
@yanchangNvidia

These changes introduce new environments:
`Isaac-PickPlace-Locomanipulation-G1-Abs-v0` and
`Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0`. These environments,
specially the locomanipulation one, feature:
1. The G1 robot with the retargeter for 3-finger dexterous hand to
operator hands tracked by Apple Vision Pro
2. A locomotion policy trained with RL that enables G1 to walk and
follow a input velocity command (vx, vy, wz, h) linear velocity in two
axes, angular velocity around its pelvis' z-axis, and height distance of
the pelvis from the feet.
3. Integration with upper body controller that is driven by Inverse
Kinematics.

Below is a diagram showing the high level controller composition. The
environments introduced have the waist controlled by IK, but it is
possible to configure it such that waist DOFs are controlled by the
lower body policy as well.
<img width="645" height="275" alt="image"
src="https://github.com/user-attachments/assets/accc4a46-dda5-4714-9a54-7812125753a2"
/>

This code has been tested through the [Mimic
pipeline](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html),
and the output dataset was used to [train a robomimic
policy](https://laughing-dollop-wom7611.pages.github.io/devel/source/overview/teleop_imitation.html#training-an-agent)
that achieves a success rate of **92% from 50 policy rollouts**.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->
Some fixes included in these changes:
1. Previously, Inverse kinematics relied on an input URDF of the robot
to load a pinocchio kinematics model. This URDF was exported from the
robot asset USD through USD to URDF export extension, however, if the
extension is not setup properly, the joint ordering can change causing
unexpected behavior due to mismatch of joint ordering between the IK
controller and that of Isaac Lab. For the G1 asset, we are for now using
a URDF asset directly instead of relying on the URDF exporter.
2. Previously, the Inverse Kinematics controller assumed that for its
kinematics model, only the joints that are being controlled by IK need
to be updated, but when combined with other controllers on other joints
this assumption breaks. Added a new kinematics configuration
`class:PinkKinematicsConfiguration` that allows for all joints of an
articulation to be updated while only a subset are controlled by IK.

<!-- As you go through the list, delete the ones that are not
applicable. -->

- New feature (non-breaking change which adds functionality)

![Locomanipulation Teacher Policy - Robomimic
Policy](https://github.com/user-attachments/assets/983b12af-f8e2-42bb-a942-92380348a843)
Robomimic policy for PickPlace trained with the Locomanipulation
environment. We achieve 92% success rate from 50 rollouts.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

⚠️ This PR depends on isaac-sim#3149  and should only be merged after that PR
<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Co-authored-by: Huihua Zhao <huihuaz@nvidia.com>
Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com>
Co-authored-by: Sergey Grizan <sgrizan@nvidia.com>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Philipp Reist <66367163+preist-nvidia@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants