Manus/hmd hand tracking

docs/source/device/index.rst

@@ -19,3 +19,4 @@ See the `Plugins directory <https://github.com/NVIDIA/IsaacTeleop/tree/main/src/
 
    trackers
    add_device
+   manus

docs/source/device/manus.rst

@@ -0,0 +1,178 @@
+..
+   SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+   SPDX-License-Identifier: Apache-2.0
+
+Manus Gloves
+============
+
+A Linux-only plugin for integrating `Manus <https://www.manus-meta.com/>`_ gloves
+into the Isaac Teleop framework. It provides full hand-joint tracking via the
+Manus SDK and injects the resulting poses into the OpenXR hand-tracking layer so
+any downstream retargeter can consume them transparently.
+
+Components
+----------
+
+- **Core library** (``manus_plugin_core``) — wraps the Manus SDK
+  (``libIsaacTeleopPluginsManus.so``) and exposes per-joint tracking data.
+- **Plugin executable** (``manus_hand_plugin``) — the main plugin binary that
+  integrates with the Teleop system via CloudXR / OpenXR.
+- **CLI tool** (``manus_hand_tracker_printer``) — a standalone diagnostic tool
+  that prints tracked joint data to the terminal for quick verification.
+
+Prerequisites
+-------------
+
+- **Linux** — x86_64 (tested on Ubuntu 22.04 / 24.04).
+- **Manus SDK** for Linux — downloaded automatically by the install script.
+- **System dependencies** — the install script prompts to install required packages.
+
+Installation
+------------
+
+Automated (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The install script handles SDK download, dependency installation, and building:
+
+.. code-block:: bash
+
+   cd src/plugins/manus
+   ./install_manus.sh
+
+The script will:
+
+1. Ask whether to install **MANUS Core Integrated** dependencies only (faster) or
+   both **Integrated and Remote** dependencies (includes gRPC, takes longer).
+2. Install the required system packages.
+3. Download MANUS SDK v3.1.1.
+4. Extract and place the SDK in the correct location.
+5. Build the plugin and the diagnostic tool
+
+Manual
+~~~~~~
+
+If you prefer to install manually:
+
+1. Download the MANUS Core SDK from
+   `MANUS Downloads <https://docs.manus-meta.com/3.1.1/Resources/>`_.
+2. Extract and place the ``ManusSDK`` folder inside ``src/plugins/manus/``, or
+   point CMake at a different path by setting ``MANUS_SDK_ROOT``.
+3. Follow the
+   `MANUS Getting Started guide for Linux <https://docs.manus-meta.com/3.1.1/Plugins/SDK/Linux/>`_
+   to install the dependencies and configure device permissions.
+
+Expected directory layout after placing the SDK:
+
+.. code-block:: text
+
+   src/plugins/manus/
+     app/
+       main.cpp
+     core/
+       manus_hand_tracking_plugin.cpp
+     inc/
+       core/
+         manus_hand_tracking_plugin.hpp
+     tools/
+       manus_hand_tracker_printer.cpp
+     ManusSDK/        <-- placed here
+       include/
+       lib/
+
+Then build from the root:
+
+.. code-block:: bash
+
+   cd ../../..  # navigate to root
+   cmake -S . -B build
+   cmake --build build --target manus_hand_plugin manus_hand_tracker_printer -j
+   cmake --install build --component manus
+
+Running the Plugin
+------------------
+
+1. Set up the CloudXR environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Source the CloudXR environment and start the runtime before running the plugin:
+
+.. code-block:: bash
+
+   export NV_CXR_RUNTIME_DIR=~/.cloudxr/run
+   export XR_RUNTIME_JSON=~/.cloudxr/openxr_cloudxr.json
+
+2. Verify with the CLI tool
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Ensure Manus Core is running and the gloves are connected, then run:
+
+.. code-block:: bash
+
+   ./build/bin/manus_hand_tracker_printer
+
+3. Run the plugin
+~~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+   ./install/plugins/manus/manus_hand_plugin
+
+Wrist Positioning — Controllers vs Optical Hand Tracking
+---------------------------------------------------------
+
+Two sources are available for positioning the Manus gloves in 3D space:
+
+- **Controller adapters** — attach Quest 3 controllers to the Manus Universal
+  Mount on the back of the glove. The controller pose drives wrist placement.
+- **Optical hand tracking** — use the HMD's built-in optical hand tracking to
+  position the hands. No physical controller adapter required.
+
+The plugin selects the source automatically at runtime: optical hand tracking is
+preferred when ``XR_MNDX_xdev_space`` is supported and the runtime reports an
+actively tracked wrist pose; otherwise it falls back to the controller pose.
+
+.. note::
+
+   When using controller adapters it is advisable to disable the HMD's automatic
+   hand-tracking–to–controller switching to avoid unexpected source changes
+   mid-session.
+
+Troubleshooting
+---------------
+
+.. list-table::
+   :widths: 40 60
+   :header-rows: 1
+
+   * - Symptom
+     - Resolution
+   * - SDK download fails
+     - Check your internet connection and re-run the install script.
+   * - Manus SDK not found at build time
+     - With manual installation, ensure ``ManusSDK`` is inside
+       ``src/plugins/manus/`` or set ``MANUS_SDK_ROOT`` to your installation path.
+   * - Manus SDK not found at runtime
+     - The build configures RPATH automatically. If you moved the SDK after
+       building, set ``LD_LIBRARY_PATH`` to its ``lib/`` directory.
+   * - No data received
+     - Ensure Manus Core is running and the gloves are connected and calibrated.
+   * - CloudXR runtime errors
+     - Make sure ``scripts/setup_cloudxr_env.sh`` has been sourced before running.
+   * - Permission denied for USB devices
+     - The install script configures udev rules. If the rules were not reloaded,
+       run:
+
+       .. code-block:: bash
+
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger
+
+       Then reconnect your Manus devices.
+
+License
+-------
+
+Source files are covered by their stated licenses (Apache-2.0). The Manus SDK is
+proprietary to Manus and is subject to its own license; it is **not** redistributed
+by this project.

scripts/run_isaac_lab.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+# Source shared CloudXR environment setup
+source scripts/setup_cloudxr_env.sh
+
+if [ ! -f "$XR_RUNTIME_JSON" ]; then
+    echo "Error: $XR_RUNTIME_JSON not found. Please run ./scripts/run_cloudxr.sh first."
+    exit 1
+fi
+
+# Run the Isaac Lab script
+if [ -z "${ISAACLAB_PATH:-}" ]; then
+    echo "Error: ISAACLAB_PATH environment variable is not set. Please set it to the path of the Isaac Lab repository."
+    exit 1
+fi
+
+if [ ! -d "$ISAACLAB_PATH" ]; then
+    echo "Error: ISAACLAB_PATH '$ISAACLAB_PATH' does not exist or is not a directory."
+    exit 1
+fi
+
+if [ ! -f "$ISAACLAB_PATH/isaaclab.sh" ]; then
+    echo "Error: Isaac Lab script not found in $ISAACLAB_PATH."
+    echo "Please make sure you have set the ISAACLAB_PATH environment variable to the path of the Isaac Lab repository."
+    exit 1
+fi
+
+# Resolve the IsaacTeleop repo root (directory containing this script's parent)
+# before cd'ing away, so we can point VIRTUAL_ENV at the correct venv.
+ISAACTELEOP_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+cd "$ISAACLAB_PATH"
+
+# Use venv_isaacteleop (has isaacsim + isaaclab), falling back to env_isaaclab.
+if [ -f "${ISAACTELEOP_ROOT}/venv_isaacteleop/bin/python" ]; then
+    export VIRTUAL_ENV="${ISAACTELEOP_ROOT}/venv_isaacteleop"
+elif [ -f "${ISAACTELEOP_ROOT}/env_isaaclab/bin/python" ]; then
+    export VIRTUAL_ENV="${ISAACTELEOP_ROOT}/env_isaaclab"
+elif [ -f "${ISAACLAB_PATH}/env_isaaclab/bin/python" ]; then
+    export VIRTUAL_ENV="${ISAACLAB_PATH}/env_isaaclab"
+else
+    unset VIRTUAL_ENV
+fi
+
+if [ -n "${ISAACSIM_PATH:-}" ] && [ -f "$ISAACSIM_PATH/setup_conda_env.sh" ]; then
+    # This is only necessary if Isaac Sim is installed via source code:
+    # https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/source_installation.html
+    if [ -n "${CONDA_PREFIX:-}" ]; then
+        echo "Setting up Isaac Sim conda environment..."
+        source "$ISAACSIM_PATH/setup_conda_env.sh"
+    fi
+fi
+
+./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
+    --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \
+    --num_envs 1 \
+    --teleop_device handtracking \
+    --device cpu \
+    --enable_pinocchio \
+    --info \
+    --xr

src/plugins/manus/README.md

@@ -15,16 +15,35 @@ This folder provides a Linux-only example of using the Manus SDK for hand tracki
 
 ## Prerequisites
 
-- **Linux** (x86_64, tested on Ubuntu 22.04)
-- **Manus SDK** for Linux x86_64
+- **Linux** (x86_64 tested on Ubuntu 22.04/24.04)
+- **Manus SDK** for Linux (automatically downloaded by install script)
+- **System dependencies**: The install script will prompt to install required packages
 
-## Getting the Manus SDK
+## Installation
 
-The Manus SDK must be downloaded separately due to licensing.
+### Automated Installation (Recommended)
 
-1. Obtain a Manus account and credentials.
-2. Download the MANUS Core SDK from [Manus Downloads](https://my.manus-meta.com/resources/downloads).
-3. Extract and place the `ManusSDK` folder inside `src/plugins/manus/`, or set the `MANUS_SDK_ROOT` environment variable to your installation path.
+Use the provided installation script which handles SDK download, dependency installation, and building:
+
+```bash
+cd src/plugins/manus
+./install_manus.sh
+```
+
+The script will:
+1. Ask whether to install MANUS Core Integrated dependencies only (faster) or both Integrated and Remote dependencies (includes gRPC, takes longer)
+2. Install required system packages
+3. Automatically download the MANUS SDK v3.1.1
+4. Extract and configure the SDK in the correct location
+5. Build the plugin
+
+### Manual Installation
+
+If you prefer to install manually:
+
+1. Download the MANUS Core SDK from [MANUS Downloads](https://docs.manus-meta.com/3.1.1/Resources/)
+2. Extract and place the `ManusSDK` folder inside `src/plugins/manus/`, or set the `MANUS_SDK_ROOT` environment variable to your installation path
+3. Follow the [MANUS Getting Started guide for Linux](https://docs.manus-meta.com/3.1.1/Plugins/SDK/Linux/) to install the dependencies and setup device permissions.
 
 Expected layout:
 ```text
@@ -43,41 +62,63 @@ src/plugins/manus/
     lib/
 ```
 
-## Build
-
-This plugin is built as part of the Isaac Teleop project. It will be automatically included if the Manus SDK is found.
-
-Run the project build from the repository root:
+4. Build from the TeleopCore root:
 
 ```bash
+cd ../../..  # Navigate to TeleopCore root
 cmake -S . -B build
-cmake --build build -j
+cmake --build build --target manus_hand_plugin -j
+cmake --install build --component manus
 ```
 
-If the SDK is not found, the build will skip the Manus plugin with a warning.
+## Running the Plugin
+
+### 1. Setup CloudXR Environment
+Before running the plugin, ensure CloudXR environment is configured:
 
-## Running the Example
+The following environment variables must be set before running either the CLI tool or the plugin (adjust paths if your CloudXR installation differs from the defaults):
+
+```bash
+export NV_CXR_RUNTIME_DIR=~/.cloudxr/run
+export XR_RUNTIME_JSON=~/.cloudxr/openxr_cloudxr.json
+```
 
-### 1. Verify with CLI Tool
+### 2. Verify with CLI Tool
 Ensure Manus Core is running and gloves are connected.
 Run the printer tool to verify data reception:
 
 ```bash
 ./build/bin/manus_hand_tracker_printer
 ```
 
-### 2. Run the Plugin
-The plugin is installed to the `install` directory.
+### 3. Run the Plugin
+The plugin is installed to the `install` directory:
 
 ```bash
 ./install/plugins/manus/manus_hand_plugin
 ```
 
+## Controller positioning vs Optical hand tracking positioning
+To position the MANUS gloves in 3D space two avenues are available:
+
+- Use the MANUS Quest 3 controller adapters to attach the Quest 3 controllers to the MANUS Universal Mount on the back of the glove.
+- Use the HMD's optical hand tracking to position the hands. 
+
+The system will switch dynamically based on the available tracking source. When using controllers it's advised to turn off hand tracking entirely or turn off automatic switching.
+
 ## Troubleshooting
 
-- **Manus SDK not found at build time**: Ensure `ManusSDK` is in `src/plugins/manus/` or `MANUS_SDK_ROOT` is set correctly.
-- **Manus SDK not found at runtime**: The build configures RPATH to find the SDK libraries. If you moved the SDK or built without RPATH support, you may need to set `LD_LIBRARY_PATH`.
-- **No data available**: Ensure Manus Core is running and gloves are properly connected and calibrated.
+- **SDK download fails**: Check your internet connection and try running the install script again
+- **Manus SDK not found at build time**: If using manual installation, ensure `ManusSDK` is in `src/plugins/manus/` or `MANUS_SDK_ROOT` is set correctly
+- **Manus SDK not found at runtime**: The CMake build configures RPATH to find the SDK libraries. If you moved the SDK, you may need to set `LD_LIBRARY_PATH`
+- **No data available**: Ensure Manus Core is running and gloves are properly connected and calibrated
+- **CloudXR runtime errors**: Make sure you've sourced `scripts/setup_cloudxr_env.sh` before running the plugin
+- **Permission denied for USB devices**: The install script configures udev rules. You may need to run:
+  ```bash
+  sudo udevadm control --reload-rules
+  sudo udevadm trigger
+  ```
+  Then reconnect your Manus devices.
 
 ## License