Adds teleop support for Unitree G1 with Inspire 5-finger hand, take PickPlace task as an example#3242
Adds teleop support for Unitree G1 with Inspire 5-finger hand, take PickPlace task as an example#3242kellyguo11 merged 15 commits into
Conversation
kellyguo11
left a comment
There was a problem hiding this comment.
@rwiltz @peterd-NV - would be great to get your review on the retargeting and pick-place task changes
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| # Copyright (c) 2022-2025, The Isaac Lab Project Developers. |
There was a problem hiding this comment.
could you clean up the headers to keep just the first section?
| """ | ||
| G1_INSPIRE_FTP_CFG = ArticulationCfg( | ||
| spawn=sim_utils.UsdFileCfg( | ||
| usd_path=f"./usecase/humanoid_teleop/g1_29dof_rev_1_0_with_inspire_hand_FTP/g1_29dof_rev_1_0_with_inspire_hand_retarget_inspire_white.usd", |
There was a problem hiding this comment.
we should get this USD uploaded to the Isaac Sim nucleus folder so that we can make it available publicly on the S3 bucket
There was a problem hiding this comment.
how different is this asset from the existing G1 on S3?
There was a problem hiding this comment.
The USD file and corresponding URDF files(left hand and right hand) for dex retargeting configuration have been uploaded to temp position Collected_g1_29dof
I am confirming with @renato Gasoto regarding the correct destination and access permissions.
This G1_29dof asset is different from the existing G1 on S3 in the following ways:
- Number of DOFs: The current G1 on S3 has 23 DOFs, while the new version has 29 DOFs.
- Composited asset: The new asset integrates the 5-finger Inspire hand, and both the robot/hand asset and merging code are provided below.
"""
Configuration for the Unitree G1 Humanoid robot with Inspire 5fingers hand.
The Unitree G1 URDF can be found here: https://github.com/unitreerobotics/unitree_ros/tree/master/robots/g1_description/g1_29dof_with_hand_rev_1_0.urdf
The Inspire hand URDF is available at: https://github.com/unitreerobotics/xr_teleoperate/tree/main/assets/inspire_hand
The merging code for the hand and robot can be found here: https://github.com/unitreerobotics/unitree_ros/blob/master/robots/g1_description/merge_g1_29dof_and_inspire_hand.ipynb,
Necessary modifications should be made to ensure the correct parent–child relationship.
"""
Note: There is a related PR3150, which includes modifications to some common files. Once PR3150 is merged, I can rebase my code to avoid any duplication. Thanks.
Signed-off-by: yami007007 <weihuaz@nvidia.com>
| """Retargeters for mapping input device data to robot commands.""" | ||
|
|
||
| from .humanoid.fourier.gr1t2_retargeter import GR1T2Retargeter, GR1T2RetargeterCfg | ||
| from .humanoid.unitree.unitree_g1_retargeter import UnitreeG1Retargeter, UnitreeG1RetargeterCfg |
There was a problem hiding this comment.
high level comment:
robot - worflow specific definition maybe more appropriate in isaaclab_tasks rather than in core. Things in core should be mostly just pure interface how robot - worflow specific definition should use them
There was a problem hiding this comment.
Yes, that makes more sense. Since we are following the GR1 convention and the code freeze is approaching, we can refactor this part in the next release?
kellyguo11
left a comment
There was a problem hiding this comment.
The gifs are quite large, can we ask Sheikh to upload them to the server similar to the other mimic gifs? https://nvidia.slack.com/archives/C07C06BV4VB/p1747662149763689. image files should be .jpg for smaller image size.
| hand_joint_names: list[str], | ||
| right_hand_config_filename: str = "unitree_hand_right_dexpilot.yml", | ||
| left_hand_config_filename: str = "unitree_hand_left_dexpilot.yml", | ||
| left_hand_urdf_path: str = "omniverse://isaac-dev.ov.nvidia.com/Projects/agile/Robots/Collected_g1_29dof/g1_29dof_rev_1_0_retarget_left_hand.urdf", |
There was a problem hiding this comment.
assets will need to be uploaded to S3.
| """ | ||
| G1_INSPIRE_FTP_CFG = ArticulationCfg( | ||
| spawn=sim_utils.UsdFileCfg( | ||
| usd_path="omniverse://isaac-dev.ov.nvidia.com/Projects/agile/Robots/Collected_g1_29dof/g1_29dof_rev_1_0_with_inspire_hand.usd", |
There was a problem hiding this comment.
asset will need to be uploaded to S3
There was a problem hiding this comment.
do we have the license for this file as well?
There was a problem hiding this comment.
Currently, there is an issue with the usd asset. Confirming with Renato.
| @@ -0,0 +1,147 @@ | |||
| # Teleoperation | |||
There was a problem hiding this comment.
can we move this to the official docs instead of having it separately in the code? imitation learning and teleoperation docs are under https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/index.html. we can also make this into a how-to guide here - https://isaac-sim.github.io/IsaacLab/main/source/how-to/index.html
There was a problem hiding this comment.
Moved the ReadMe to docs/source/how-to/cloudxr_teleoperation.rst.
| body_pos_w = env.scene["robot"].data.body_pos_w | ||
| left_eef_idx = env.scene["robot"].data.body_names.index("left_hand_roll_link") | ||
| right_eef_idx = env.scene["robot"].data.body_names.index("right_hand_roll_link") | ||
| if env.unwrapped.cfg.robot_name == "Unitree_G1": |
There was a problem hiding this comment.
would be good to avoid the ifs by robot name. can we make the body name a parameter instead that can be define through the config?
There was a problem hiding this comment.
Updated, use parameter instead of if statements.
| ) -> torch.Tensor: | ||
| hand_joint_states = env.scene["robot"].data.joint_pos[:, -22:] # Hand joints are last 22 entries of joint state | ||
|
|
||
| if env.unwrapped.cfg.robot_name == "Unitree_G1": |
There was a problem hiding this comment.
would be good to avoid the if statement here as well
There was a problem hiding this comment.
Updated: use a parameter instead of if statements.
| # Get right wrist position relative to environment origin | ||
| robot_body_pos_w = env.scene["robot"].data.body_pos_w | ||
| right_eef_idx = env.scene["robot"].data.body_names.index("right_hand_roll_link") | ||
| if env.unwrapped.cfg.robot_name == "Unitree_G1": |
Deleted the gifs, moved the ReadMe to docs/source/how-to/cloudxr_teleoperation.rst. |
| hand_joint_names: list[str] | None = None # List of robot hand joint names | ||
|
|
||
|
|
||
| class UnitreeG1Retargeter(RetargeterBase): |
There was a problem hiding this comment.
I think we need to rebase this change on top of: #3150 and rename both of the retargeters so they are clear which is the 5 finger hand and which is the 3 finger hand https://github.com/isaac-sim/IsaacLab/pull/3150/files#diff-736ef5d8d689c193e9a8d81c7348d635659fb8fc4a0e28c9447eec5e986d0c5cR33
There was a problem hiding this comment.
That makes more sense. Updated.
There was a problem hiding this comment.
Move source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_retargeter.py -> source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/inspire/g1_upper_body_retargeter.py
There was a problem hiding this comment.
That makes more sense. Updated.
There was a problem hiding this comment.
Move source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/g1_dex_retargeting_utils.py
-> source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/inspire/g1_dex_retargeting_utils.py
There was a problem hiding this comment.
Move source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/data/configs/dex-retargeting/unitree_hand_left_dexpilot.yml -> source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/inspire/data/configs/dex-retargeting/unitree_hand_left_dexpilot.yml
There was a problem hiding this comment.
Move source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/data/configs/dex-retargeting/unitree_hand_right_dexpilot.yml -> source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/unitree/inspire/data/configs/dex-retargeting/unitree_hand_right_dexpilot.yml
| position_cost=8.0, # [cost] / [m] | ||
| orientation_cost=1.0, # [cost] / [rad] | ||
| lm_damping=10, # dampening for solver for step jumps | ||
| lm_damping=12, # dampening for solver for step jumps |
There was a problem hiding this comment.
@michaellin6 is this a concern for any downstream datasets or workflows?
There was a problem hiding this comment.
Not sure why this change is needed, but at least it passes my test so it should be ok.
…n/pick_place/pickplace_unitree_g1_inspire_hand_env_cfg.py Signed-off-by: Kelly Guo <kellyg@nvidia.com>
…n/pick_place/pickplace_unitree_g1_inspire_hand_env_cfg.py Signed-off-by: Kelly Guo <kellyg@nvidia.com>
# Description Fix: teleop G1 with Inspire hand (PR #3242), modifications includes: - Correct the USD asset path of G1_INSPIRE_FTP_CFG - Increase arm damping for stability using the new pink_ik wrapper ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots - Only modified the `unitree.py` file. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
) # Description Fix: teleop G1 with Inspire hand (PR #3242), add the Isaac-PickPlace-G1-InspireFTP-Abs-v0 task into environment docs. ## Type of change - Documentation update ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
…ickPlace task as an example (isaac-sim#3242) # Description Teleop support for Unitree G1 with Inspire 5-finger hand, take PickPlace task as an example. # Type of change - New feature (non-breaking change which adds functionality) ## Changes: -Preparation of the humanoid robot asset: Unitree G1 (29 DOF) with Inspire 5-finger hand -Configuration of the robot’s articulation -Retargeting setup for the humanoid robot -Creation of the manipulation task environment: pickplace_unitree_g1_inspire_hand_env_cfg.py -Updated the how-to tutorial of [cloudxr_teleoperation.rst](https://github.com/isaac-sim/IsaacLab/pull/3242/files#diff-3896784d4dbd3e2106adae8f4ab65530a3bb87430cf8d15463aff37f3a975fec). Below is a diagram showing the robot asset and the workflow in teleop.  # Test Steps 1. Run the teleop task with: python scripts/tools/record_demos.py \ --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \ --teleop_device handtracking \ --enable_pinocchio 2. Replay the collected demos with: python scripts/tools/replay_demos.py \ --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \ --enable_pinocchio \ --dataset_file datasets/dataset.hdf5 ## Screenshots Please attach before and after screenshots of the change if applicable.  ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: yami007007 <weihuaz@nvidia.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Weihua Zhang <weihuaz@weihuaz-mlt.client.nvidia.com>
# Description Fix: teleop G1 with Inspire hand (PR isaac-sim#3242), modifications includes: - Correct the USD asset path of G1_INSPIRE_FTP_CFG - Increase arm damping for stability using the new pink_ik wrapper ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots - Only modified the `unitree.py` file. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
…aac-sim#3479) # Description Fix: teleop G1 with Inspire hand (PR isaac-sim#3242), add the Isaac-PickPlace-G1-InspireFTP-Abs-v0 task into environment docs. ## Type of change - Documentation update ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
…ickPlace task as an example (isaac-sim#3242) Teleop support for Unitree G1 with Inspire 5-finger hand, take PickPlace task as an example. - New feature (non-breaking change which adds functionality) -Preparation of the humanoid robot asset: Unitree G1 (29 DOF) with Inspire 5-finger hand -Configuration of the robot’s articulation -Retargeting setup for the humanoid robot -Creation of the manipulation task environment: pickplace_unitree_g1_inspire_hand_env_cfg.py -Updated the how-to tutorial of [cloudxr_teleoperation.rst](https://github.com/isaac-sim/IsaacLab/pull/3242/files#diff-3896784d4dbd3e2106adae8f4ab65530a3bb87430cf8d15463aff37f3a975fec). Below is a diagram showing the robot asset and the workflow in teleop.  1. Run the teleop task with: python scripts/tools/record_demos.py \ --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \ --teleop_device handtracking \ --enable_pinocchio 2. Replay the collected demos with: python scripts/tools/replay_demos.py \ --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \ --enable_pinocchio \ --dataset_file datasets/dataset.hdf5 Please attach before and after screenshots of the change if applicable.  - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: yami007007 <weihuaz@nvidia.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Weihua Zhang <weihuaz@weihuaz-mlt.client.nvidia.com>
# Description Fix: teleop G1 with Inspire hand (PR isaac-sim#3242), modifications includes: - Correct the USD asset path of G1_INSPIRE_FTP_CFG - Increase arm damping for stability using the new pink_ik wrapper ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots - Only modified the `unitree.py` file. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
…aac-sim#3479) Fix: teleop G1 with Inspire hand (PR isaac-sim#3242), add the Isaac-PickPlace-G1-InspireFTP-Abs-v0 task into environment docs. - Documentation update - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
Description
Teleop support for Unitree G1 with Inspire 5-finger hand, take PickPlace task as an example.
Type of change
Changes:
-Preparation of the humanoid robot asset: Unitree G1 (29 DOF) with Inspire 5-finger hand
-Configuration of the robot’s articulation
-Retargeting setup for the humanoid robot
-Creation of the manipulation task environment: pickplace_unitree_g1_inspire_hand_env_cfg.py
-Updated the how-to tutorial of cloudxr_teleoperation.rst.
Below is a diagram showing the robot asset and the workflow in teleop.

Test Steps
Run the teleop task with:
python scripts/tools/record_demos.py
--task Isaac-PickPlace-G1-InspireFTP-Abs-v0
--teleop_device handtracking
--enable_pinocchio
Replay the collected demos with:
python scripts/tools/replay_demos.py
--task Isaac-PickPlace-G1-InspireFTP-Abs-v0
--enable_pinocchio
--dataset_file datasets/dataset.hdf5
Screenshots
Please attach before and after screenshots of the change if applicable.

Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there