Adds Locomanipulation Environment with G1 for Mimic workflow

CONTRIBUTORS.md

@@ -111,6 +111,7 @@ Guidelines for modifications:
 * Özhan Özen
 * Patrick Yin
 * Peter Du
+* Philipp Reist
 * Pulkit Goyal
 * Qian Wan
 * Qinxi Yu
@@ -120,6 +121,7 @@ Guidelines for modifications:
 * Ritvik Singh
 * Rosario Scalise
 * Ryley McCarroll
+* Sergey Grizan
 * Shafeef Omar
 * Shaoshu Su
 * Shaurya Dewan

docs/source/overview/imitation-learning/teleop_imitation.rst

@@ -140,7 +140,7 @@ Pre-recorded demonstrations
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 We provide a pre-recorded ``dataset.hdf5`` containing 10 human demonstrations for ``Isaac-Stack-Cube-Franka-IK-Rel-v0``
-`here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/franka_stack_datasets/dataset.hdf5>`__.
+here: `[Franka Dataset] <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/franka_stack_datasets/dataset.hdf5>`__.
 This dataset may be downloaded and used in the remaining tutorial steps if you do not wish to collect your own demonstrations.
 
 .. note::
@@ -451,7 +451,7 @@ Generate the dataset
 ^^^^^^^^^^^^^^^^^^^^
 
 If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset ``dataset_annotated_gr1.hdf5`` from
-`here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/pick_place_datasets/dataset_annotated_gr1.hdf5>`__.
+here: `[Annotated GR1 Dataset] <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/pick_place_datasets/dataset_annotated_gr1_v01.hdf5>`_.
 Place the file under ``IsaacLab/datasets`` and run the following command to generate a new dataset with 1000 demonstrations.
 
 .. code:: bash
@@ -508,13 +508,132 @@ Visualize the results of the trained policy by running the following command, us
    The trained policy performing the pick and place task in Isaac Lab.
 
 
-Demo 2: Visuomotor Policy for a Humanoid Robot
+Demo 2: Data Generation and Policy Training for Humanoid Robot Locomanipulation with Unitree G1
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In this demo, we showcase the integration of locomotion and manipulation capabilities within a single humanoid robot system.
+This locomanipulation environment enables data collection for complex tasks that combine navigation and object manipulation.
+The demonstration follows a multi-step process: first, it generates pick and place tasks similar to Demo 1, then introduces
+a navigation component that uses specialized scripts to generate scenes where the humanoid robot must move from point A to point B.
+The robot picks up an object at the initial location (point A) and places it at the target destination (point B).
+
+.. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/locomanipulation-g-1_steering_wheel_pick_place.gif
+   :width: 100%
+   :align: center
+   :alt: G1 humanoid robot with locomanipulation performing a pick and place task
+   :figclass: align-center
+
+Generate the manipulation dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The same data generation and policy training steps from Demo 1.0 can be applied to the G1 humanoid robot with locomanipulation capabilities.
+This demonstration shows how to train a G1 robot to perform pick and place tasks with full-body locomotion and manipulation.
+
+The process follows the same workflow as Demo 1.0, but uses the ``Isaac-PickPlace-Locomanipulation-G1-Abs-v0`` task environment.
+
+Follow the same data collection, annotation, and generation process as demonstrated in Demo 1.0, but adapted for the G1 locomanipulation task.
+
+.. hint::
+
+   If desired, data collection and annotation can be done using the same commands as the prior examples for validation of the dataset.
+
+   The G1 robot with locomanipulation capabilities combines full-body locomotion with manipulation to perform pick and place tasks.
+
+   **Note that the following commands are only for your reference and dataset validation purposes - they are not required for this demo.**
+
+   To collect demonstrations:
+
+   .. code:: bash
+
+      ./isaaclab.sh -p scripts/tools/record_demos.py \
+      --device cpu \
+      --task Isaac-PickPlace-Locomanipulation-G1-Abs-v0 \
+      --teleop_device handtracking \
+      --dataset_file ./datasets/dataset_g1_locomanip.hdf5 \
+      --num_demos 5 --enable_pinocchio
+
+   You can replay the collected demonstrations by running:
+
+   .. code:: bash
+
+      ./isaaclab.sh -p scripts/tools/replay_demos.py \
+      --device cpu \
+      --task Isaac-PickPlace-Locomanipulation-G1-Abs-v0 \
+      --dataset_file ./datasets/dataset_g1_locomanip.hdf5 --enable_pinocchio
+
+   To annotate the demonstrations:
+
+   .. code:: bash
+
+      ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
+      --device cpu \
+      --task Isaac-PickPlace-Locomanipulation-G1-Abs-Mimic-v0 \
+      --input_file ./datasets/dataset_g1_locomanip.hdf5 \
+      --output_file ./datasets/dataset_annotated_g1_locomanip.hdf5 --enable_pinocchio
+
+
+If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset ``dataset_annotated_g1_locomanip.hdf5`` from
+here: `[Annotated G1 Dataset] <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/locomanip_datasets/dataset_annotated_g1_locomanip_v01.hdf5>`_.
+Place the file under ``IsaacLab/datasets`` and run the following command to generate a new dataset with 1000 demonstrations.
+
+.. code:: bash
+
+   ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
+   --device cpu --headless --num_envs 20 --generation_num_trials 1000 --enable_pinocchio \
+   --input_file ./datasets/dataset_annotated_g1_locomanip.hdf5 --output_file ./datasets/generated_dataset_g1_locomanip.hdf5
+
+
+Train a manipulation-only policy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+At this point you can train a policy that only performs manipulation tasks using the generated dataset:
+
+.. code:: bash
+
+   ./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py \
+   --task Isaac-PickPlace-Locomanipulation-G1-Abs-v0 --algo bc \
+   --normalize_training_actions \
+   --dataset ./datasets/generated_dataset_g1_locomanip.hdf5
+
+Visualize the results
+^^^^^^^^^^^^^^^^^^^^^
+
+Visualize the trained policy performance:
+
+.. code:: bash
+
+   ./isaaclab.sh -p scripts/imitation_learning/robomimic/play.py \
+   --device cpu \
+   --enable_pinocchio \
+   --task Isaac-PickPlace-Locomanipulation-G1-Abs-v0 \
+   --num_rollouts 50 \
+   --horizon 400 \
+   --norm_factor_min <NORM_FACTOR_MIN> \
+   --norm_factor_max <NORM_FACTOR_MAX> \
+   --checkpoint /PATH/TO/desired_model_checkpoint.pth
+
+.. note::
+   Change the ``NORM_FACTOR`` in the above command with the values generated in the training step.
+
+.. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/locomanipulation-g-1_steering_wheel_pick_place.gif
+   :width: 100%
+   :align: center
+   :alt: G1 humanoid robot performing a pick and place task
+   :figclass: align-center
+
+   The trained policy performing the pick and place task in Isaac Lab.
+
+Generate the dataset with manipulation and point-to-point navigation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+Demo 3: Visuomotor Policy for a Humanoid Robot
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Download the Dataset
 ^^^^^^^^^^^^^^^^^^^^
 
-Download the pre-generated dataset from `here <https://download.isaacsim.omniverse.nvidia.com/isaaclab/dataset/generated_dataset_gr1_nut_pouring.hdf5>`__ and place it under ``IsaacLab/datasets/generated_dataset_gr1_nut_pouring.hdf5``.
+Download the pre-generated dataset from here: `[Generated GR1 Dataset] <https://download.isaacsim.omniverse.nvidia.com/isaaclab/dataset/generated_dataset_gr1_nut_pouring.hdf5>`__ and place it under ``IsaacLab/datasets/generated_dataset_gr1_nut_pouring.hdf5``.
 The dataset contains 1000 demonstrations of a humanoid robot performing a pouring/placing task that was
 generated using Isaac Lab Mimic for the ``Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0`` task.
 

scripts/environments/teleoperation/teleop_se3_agent.py

@@ -66,6 +66,7 @@
 from isaaclab_tasks.utils import parse_env_cfg
 
 if args_cli.enable_pinocchio:
+    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
     import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401
 
 

scripts/imitation_learning/robomimic/play.py

@@ -70,6 +70,7 @@
 
 if args_cli.enable_pinocchio:
     import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401
+    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
 
 from isaaclab_tasks.utils import parse_env_cfg
 

scripts/imitation_learning/robomimic/train.py

@@ -84,6 +84,7 @@
 
 # Isaac Lab imports (needed so that environment is registered)
 import isaaclab_tasks  # noqa: F401
+import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
 import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401
 
 

scripts/tools/record_demos.py

@@ -98,6 +98,7 @@
 
 if args_cli.enable_pinocchio:
     import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401
+    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
 
 from collections.abc import Callable
 

scripts/tools/replay_demos.py

@@ -66,6 +66,7 @@
 
 if args_cli.enable_pinocchio:
     import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401
+    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
 
 import isaaclab_tasks  # noqa: F401
 from isaaclab_tasks.utils.parse_cfg import parse_env_cfg

source/isaaclab/config/extension.toml

@@ -1,7 +1,7 @@
 [package]
 
 # Note: Semantic Versioning is used: https://semver.org/
-version = "0.45.15"
+version = "0.45.16"
 
 # Description
 title = "Isaac Lab framework for Robot Learning"

source/isaaclab/docs/CHANGELOG.rst

@@ -1,6 +1,16 @@
 Changelog
 ---------
 
+0.45.16 (2025-08-27)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added teleoperation environments for Unitree G1. This includes an environment with lower body fixed and upper body
+  controlled by IK, and an environment with the lower body controlled by a policy and the upper body controlled by IK.
+
+
 0.45.15 (2025-09-05)
 ~~~~~~~~~~~~~~~~~~~~
 

source/isaaclab/isaaclab/assets/articulation/articulation_data.py

@@ -151,8 +151,9 @@ def update(self, dt: float):
     default_inertia: torch.Tensor = None
     """Default inertia for all the bodies in the articulation. Shape is (num_instances, num_bodies, 9).
 
-    The inertia is the inertia tensor relative to the center of mass frame. The values are stored in
-    the order :math:`[I_{xx}, I_{xy}, I_{xz}, I_{yx}, I_{yy}, I_{yz}, I_{zx}, I_{zy}, I_{zz}]`.
+    The inertia tensor should be given with respect to the center of mass, expressed in the articulation links' actor frame.
+    The values are stored in the order :math:`[I_{xx}, I_{yx}, I_{zx}, I_{xy}, I_{yy}, I_{zy}, I_{xz}, I_{yz}, I_{zz}]`.
+    However, due to the symmetry of inertia tensors, row- and column-major orders are equivalent.
 
     This quantity is parsed from the USD schema at the time of initialization.
     """

source/isaaclab/isaaclab/assets/rigid_object/rigid_object_data.py

@@ -112,8 +112,11 @@ def update(self, dt: float):
     default_inertia: torch.Tensor = None
     """Default inertia tensor read from the simulation. Shape is (num_instances, 9).
 
-    The inertia is the inertia tensor relative to the center of mass frame. The values are stored in
-    the order :math:`[I_{xx}, I_{xy}, I_{xz}, I_{yx}, I_{yy}, I_{yz}, I_{zx}, I_{zy}, I_{zz}]`.
+    The inertia tensor should be given with respect to the center of mass, expressed in the rigid body's actor frame.
+    The values are stored in the order :math:`[I_{xx}, I_{yx}, I_{zx}, I_{xy}, I_{yy}, I_{zy}, I_{xz}, I_{yz}, I_{zz}]`.
+    However, due to the symmetry of inertia tensors, row- and column-major orders are equivalent.
+
+    This quantity is parsed from the USD schema at the time of initialization.
     """
 
     ##

source/isaaclab/isaaclab/assets/rigid_object_collection/rigid_object_collection_data.py

@@ -118,8 +118,11 @@ def update(self, dt: float):
     default_inertia: torch.Tensor = None
     """Default object inertia tensor read from the simulation. Shape is (num_instances, num_objects, 9).
 
-    The inertia is the inertia tensor relative to the center of mass frame. The values are stored in
-    the order :math:`[I_{xx}, I_{xy}, I_{xz}, I_{yx}, I_{yy}, I_{yz}, I_{zx}, I_{zy}, I_{zz}]`.
+    The inertia tensor should be given with respect to the center of mass, expressed in the rigid body's actor frame.
+    The values are stored in the order :math:`[I_{xx}, I_{yx}, I_{zx}, I_{xy}, I_{yy}, I_{zy}, I_{xz}, I_{yz}, I_{zz}]`.
+    However, due to the symmetry of inertia tensors, row- and column-major orders are equivalent.
+
+    This quantity is parsed from the USD schema at the time of initialization.
     """
 
     ##

source/isaaclab/isaaclab/controllers/pink_ik/local_frame_task.py

@@ -0,0 +1,116 @@
+# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+import numpy as np
+from collections.abc import Sequence
+
+import pinocchio as pin
+from pink.tasks.frame_task import FrameTask
+
+from .pink_kinematics_configuration import PinkKinematicsConfiguration
+
+
+class LocalFrameTask(FrameTask):
+    """
+    A task that computes error in a local (custom) frame.
+    Inherits from FrameTask but overrides compute_error.
+    """
+
+    def __init__(
+        self,
+        frame: str,
+        base_link_frame_name: str,
+        position_cost: float | Sequence[float],
+        orientation_cost: float | Sequence[float],
+        lm_damping: float = 0.0,
+        gain: float = 1.0,
+    ):
+        """
+        Initialize the LocalFrameTask with configuration.
+
+        This task computes pose errors in a local (custom) frame rather than the world frame,
+        allowing for more flexible control strategies where the reference frame can be
+        specified independently.
+
+        Args:
+            frame: Name of the frame to control (end-effector or target frame).
+            base_link_frame_name: Name of the base link frame used as reference frame
+                for computing transforms and errors.
+            position_cost: Cost weight(s) for position error. Can be a single float
+                for uniform weighting or a sequence of 3 floats for per-axis weighting.
+            orientation_cost: Cost weight(s) for orientation error. Can be a single float
+                for uniform weighting or a sequence of 3 floats for per-axis weighting.
+            lm_damping: Levenberg-Marquardt damping factor for numerical stability.
+                Defaults to 0.0 (no damping).
+            gain: Task gain factor that scales the overall task contribution.
+                Defaults to 1.0.
+        """
+        super().__init__(frame, position_cost, orientation_cost, lm_damping, gain)
+        self.base_link_frame_name = base_link_frame_name
+        self.transform_target_to_base = None
+
+    def set_target(self, transform_target_to_base: pin.SE3) -> None:
+        """Set task target pose in the world frame.
+
+        Args:
+            transform_target_to_world: Transform from the task target frame to
+                the world frame.
+        """
+        self.transform_target_to_base = transform_target_to_base.copy()
+
+    def set_target_from_configuration(self, configuration: PinkKinematicsConfiguration) -> None:
+        """Set task target pose from a robot configuration.
+
+        Args:
+            configuration: Robot configuration.
+        """
+        if not isinstance(configuration, PinkKinematicsConfiguration):
+            raise ValueError("configuration must be a PinkKinematicsConfiguration")
+        self.set_target(configuration.get_transform(self.frame, self.base_link_frame_name))
+
+    def compute_error(self, configuration: PinkKinematicsConfiguration) -> np.ndarray:
+        """
+        Compute the error between current and target pose in a local frame.
+        """
+        if not isinstance(configuration, PinkKinematicsConfiguration):
+            raise ValueError("configuration must be a PinkKinematicsConfiguration")
+        if self.transform_target_to_base is None:
+            raise ValueError(f"no target set for frame '{self.frame}'")
+
+        transform_frame_to_base = configuration.get_transform(self.frame, self.base_link_frame_name)
+        transform_target_to_frame = transform_frame_to_base.actInv(self.transform_target_to_base)
+
+        error_in_frame: np.ndarray = pin.log(transform_target_to_frame).vector
+        return error_in_frame
+
+    def compute_jacobian(self, configuration: PinkKinematicsConfiguration) -> np.ndarray:
+        r"""Compute the frame task Jacobian.
+
+        The task Jacobian :math:`J(q) \in \mathbb{R}^{6 \times n_v}` is the
+        derivative of the task error :math:`e(q) \in \mathbb{R}^6` with respect
+        to the configuration :math:`q`. The formula for the frame task is:
+
+        .. math::
+
+            J(q) = -\text{Jlog}_6(T_{tb}) {}_b J_{0b}(q)
+
+        The derivation of the formula for this Jacobian is detailed in
+        [Caron2023]_. See also
+        :func:`pink.tasks.task.Task.compute_jacobian` for more context on task
+        Jacobians.
+
+        Args:
+            configuration: Robot configuration :math:`q`.
+
+        Returns:
+            Jacobian matrix :math:`J`, expressed locally in the frame.
+        """
+        if self.transform_target_to_base is None:
+            raise Exception(f"no target set for frame '{self.frame}'")
+        transform_frame_to_base = configuration.get_transform(self.frame, self.base_link_frame_name)
+        transform_frame_to_target = self.transform_target_to_base.actInv(transform_frame_to_base)
+        jacobian_in_frame = configuration.get_frame_jacobian(self.frame)
+        J = -pin.Jlog6(transform_frame_to_target) @ jacobian_in_frame
+        return J