diff --git a/docs/source/device/index.rst b/docs/source/device/index.rst index 67e03b5c4..88a3e7f8f 100644 --- a/docs/source/device/index.rst +++ b/docs/source/device/index.rst @@ -19,3 +19,4 @@ See the `Plugins directory `_ gloves +into the Isaac Teleop framework. It provides full hand-joint tracking via the +Manus SDK and injects the resulting poses into the OpenXR hand-tracking layer so +any downstream retargeter can consume them transparently. + +Components +---------- + +- **Core library** (``manus_plugin_core``) — wraps the Manus SDK + (``libIsaacTeleopPluginsManus.so``) and exposes per-joint tracking data. +- **Plugin executable** (``manus_hand_plugin``) — the main plugin binary that + integrates with the Teleop system via CloudXR / OpenXR. +- **CLI tool** (``manus_hand_tracker_printer``) — a standalone diagnostic tool + that prints tracked joint data to the terminal for quick verification. + +Prerequisites +------------- + +- **Linux** — x86_64 (tested on Ubuntu 22.04 / 24.04). +- **Manus SDK** for Linux — downloaded automatically by the install script. +- **System dependencies** — the install script prompts to install required packages. + +Installation +------------ + +Automated (recommended) +~~~~~~~~~~~~~~~~~~~~~~~~ + +The install script handles SDK download, dependency installation, and building: + +.. code-block:: bash + + cd src/plugins/manus + ./install_manus.sh + +The script will: + +1. Ask whether to install **MANUS Core Integrated** dependencies only (faster) or + both **Integrated and Remote** dependencies (includes gRPC, takes longer). +2. Install the required system packages. +3. Download MANUS SDK v3.1.1. +4. Extract and place the SDK in the correct location. +5. Build the plugin and the diagnostic tool + +Manual +~~~~~~ + +If you prefer to install manually: + +1. Download the MANUS Core SDK from + `MANUS Downloads `_. +2. Extract and place the ``ManusSDK`` folder inside ``src/plugins/manus/``, or + point CMake at a different path by setting ``MANUS_SDK_ROOT``. +3. Follow the + `MANUS Getting Started guide for Linux `_ + to install the dependencies and configure device permissions. + +Expected directory layout after placing the SDK: + +.. code-block:: text + + src/plugins/manus/ + app/ + main.cpp + core/ + manus_hand_tracking_plugin.cpp + inc/ + core/ + manus_hand_tracking_plugin.hpp + tools/ + manus_hand_tracker_printer.cpp + ManusSDK/ <-- placed here + include/ + lib/ + +Then build from the root: + +.. code-block:: bash + + cd ../../.. # navigate to root + cmake -S . -B build + cmake --build build --target manus_hand_plugin manus_hand_tracker_printer -j + cmake --install build --component manus + +Running the Plugin +------------------ + +1. Set up the CloudXR environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Source the CloudXR environment and start the runtime before running the plugin: + +.. code-block:: bash + + export NV_CXR_RUNTIME_DIR=~/.cloudxr/run + export XR_RUNTIME_JSON=~/.cloudxr/openxr_cloudxr.json + +2. Verify with the CLI tool +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Ensure Manus Core is running and the gloves are connected, then run: + +.. code-block:: bash + + ./build/bin/manus_hand_tracker_printer + +3. Run the plugin +~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + ./install/plugins/manus/manus_hand_plugin + +Wrist Positioning — Controllers vs Optical Hand Tracking +--------------------------------------------------------- + +Two sources are available for positioning the Manus gloves in 3D space: + +- **Controller adapters** — attach Quest 3 controllers to the Manus Universal + Mount on the back of the glove. The controller pose drives wrist placement. +- **Optical hand tracking** — use the HMD's built-in optical hand tracking to + position the hands. No physical controller adapter required. + +The plugin selects the source automatically at runtime: optical hand tracking is +preferred when ``XR_MNDX_xdev_space`` is supported and the runtime reports an +actively tracked wrist pose; otherwise it falls back to the controller pose. + +.. note:: + + When using controller adapters it is advisable to disable the HMD's automatic + hand-tracking–to–controller switching to avoid unexpected source changes + mid-session. + +Troubleshooting +--------------- + +.. list-table:: + :widths: 40 60 + :header-rows: 1 + + * - Symptom + - Resolution + * - SDK download fails + - Check your internet connection and re-run the install script. + * - Manus SDK not found at build time + - With manual installation, ensure ``ManusSDK`` is inside + ``src/plugins/manus/`` or set ``MANUS_SDK_ROOT`` to your installation path. + * - Manus SDK not found at runtime + - The build configures RPATH automatically. If you moved the SDK after + building, set ``LD_LIBRARY_PATH`` to its ``lib/`` directory. + * - No data received + - Ensure Manus Core is running and the gloves are connected and calibrated. + * - CloudXR runtime errors + - Make sure ``scripts/setup_cloudxr_env.sh`` has been sourced before running. + * - Permission denied for USB devices + - The install script configures udev rules. If the rules were not reloaded, + run: + + .. code-block:: bash + + sudo udevadm control --reload-rules + sudo udevadm trigger + + Then reconnect your Manus devices. + +License +------- + +Source files are covered by their stated licenses (Apache-2.0). The Manus SDK is +proprietary to Manus and is subject to its own license; it is **not** redistributed +by this project. diff --git a/scripts/run_isaac_lab.sh b/scripts/run_isaac_lab.sh new file mode 100755 index 000000000..6e586c2c3 --- /dev/null +++ b/scripts/run_isaac_lab.sh @@ -0,0 +1,66 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +# Source shared CloudXR environment setup +source scripts/setup_cloudxr_env.sh + +if [ ! -f "$XR_RUNTIME_JSON" ]; then + echo "Error: $XR_RUNTIME_JSON not found. Please run ./scripts/run_cloudxr.sh first." + exit 1 +fi + +# Run the Isaac Lab script +if [ -z "${ISAACLAB_PATH:-}" ]; then + echo "Error: ISAACLAB_PATH environment variable is not set. Please set it to the path of the Isaac Lab repository." + exit 1 +fi + +if [ ! -d "$ISAACLAB_PATH" ]; then + echo "Error: ISAACLAB_PATH '$ISAACLAB_PATH' does not exist or is not a directory." + exit 1 +fi + +if [ ! -f "$ISAACLAB_PATH/isaaclab.sh" ]; then + echo "Error: Isaac Lab script not found in $ISAACLAB_PATH." + echo "Please make sure you have set the ISAACLAB_PATH environment variable to the path of the Isaac Lab repository." + exit 1 +fi + +# Resolve the IsaacTeleop repo root (directory containing this script's parent) +# before cd'ing away, so we can point VIRTUAL_ENV at the correct venv. +ISAACTELEOP_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +cd "$ISAACLAB_PATH" + +# Use venv_isaacteleop (has isaacsim + isaaclab), falling back to env_isaaclab. +if [ -f "${ISAACTELEOP_ROOT}/venv_isaacteleop/bin/python" ]; then + export VIRTUAL_ENV="${ISAACTELEOP_ROOT}/venv_isaacteleop" +elif [ -f "${ISAACTELEOP_ROOT}/env_isaaclab/bin/python" ]; then + export VIRTUAL_ENV="${ISAACTELEOP_ROOT}/env_isaaclab" +elif [ -f "${ISAACLAB_PATH}/env_isaaclab/bin/python" ]; then + export VIRTUAL_ENV="${ISAACLAB_PATH}/env_isaaclab" +else + unset VIRTUAL_ENV +fi + +if [ -n "${ISAACSIM_PATH:-}" ] && [ -f "$ISAACSIM_PATH/setup_conda_env.sh" ]; then + # This is only necessary if Isaac Sim is installed via source code: + # https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/source_installation.html + if [ -n "${CONDA_PREFIX:-}" ]; then + echo "Setting up Isaac Sim conda environment..." + source "$ISAACSIM_PATH/setup_conda_env.sh" + fi +fi + +./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \ + --task Isaac-PickPlace-G1-InspireFTP-Abs-v0 \ + --num_envs 1 \ + --teleop_device handtracking \ + --device cpu \ + --enable_pinocchio \ + --info \ + --xr diff --git a/src/plugins/manus/README.md b/src/plugins/manus/README.md index dddb299a0..c5ccbf4d5 100644 --- a/src/plugins/manus/README.md +++ b/src/plugins/manus/README.md @@ -15,16 +15,35 @@ This folder provides a Linux-only example of using the Manus SDK for hand tracki ## Prerequisites -- **Linux** (x86_64, tested on Ubuntu 22.04) -- **Manus SDK** for Linux x86_64 +- **Linux** (x86_64 tested on Ubuntu 22.04/24.04) +- **Manus SDK** for Linux (automatically downloaded by install script) +- **System dependencies**: The install script will prompt to install required packages -## Getting the Manus SDK +## Installation -The Manus SDK must be downloaded separately due to licensing. +### Automated Installation (Recommended) -1. Obtain a Manus account and credentials. -2. Download the MANUS Core SDK from [Manus Downloads](https://my.manus-meta.com/resources/downloads). -3. Extract and place the `ManusSDK` folder inside `src/plugins/manus/`, or set the `MANUS_SDK_ROOT` environment variable to your installation path. +Use the provided installation script which handles SDK download, dependency installation, and building: + +```bash +cd src/plugins/manus +./install_manus.sh +``` + +The script will: +1. Ask whether to install MANUS Core Integrated dependencies only (faster) or both Integrated and Remote dependencies (includes gRPC, takes longer) +2. Install required system packages +3. Automatically download the MANUS SDK v3.1.1 +4. Extract and configure the SDK in the correct location +5. Build the plugin + +### Manual Installation + +If you prefer to install manually: + +1. Download the MANUS Core SDK from [MANUS Downloads](https://docs.manus-meta.com/3.1.1/Resources/) +2. Extract and place the `ManusSDK` folder inside `src/plugins/manus/`, or set the `MANUS_SDK_ROOT` environment variable to your installation path +3. Follow the [MANUS Getting Started guide for Linux](https://docs.manus-meta.com/3.1.1/Plugins/SDK/Linux/) to install the dependencies and setup device permissions. Expected layout: ```text @@ -43,22 +62,28 @@ src/plugins/manus/ lib/ ``` -## Build - -This plugin is built as part of the Isaac Teleop project. It will be automatically included if the Manus SDK is found. - -Run the project build from the repository root: +4. Build from the TeleopCore root: ```bash +cd ../../.. # Navigate to TeleopCore root cmake -S . -B build -cmake --build build -j +cmake --build build --target manus_hand_plugin -j +cmake --install build --component manus ``` -If the SDK is not found, the build will skip the Manus plugin with a warning. +## Running the Plugin + +### 1. Setup CloudXR Environment +Before running the plugin, ensure CloudXR environment is configured: -## Running the Example +The following environment variables must be set before running either the CLI tool or the plugin (adjust paths if your CloudXR installation differs from the defaults): + +```bash +export NV_CXR_RUNTIME_DIR=~/.cloudxr/run +export XR_RUNTIME_JSON=~/.cloudxr/openxr_cloudxr.json +``` -### 1. Verify with CLI Tool +### 2. Verify with CLI Tool Ensure Manus Core is running and gloves are connected. Run the printer tool to verify data reception: @@ -66,18 +91,34 @@ Run the printer tool to verify data reception: ./build/bin/manus_hand_tracker_printer ``` -### 2. Run the Plugin -The plugin is installed to the `install` directory. +### 3. Run the Plugin +The plugin is installed to the `install` directory: ```bash ./install/plugins/manus/manus_hand_plugin ``` +## Controller positioning vs Optical hand tracking positioning +To position the MANUS gloves in 3D space two avenues are available: + +- Use the MANUS Quest 3 controller adapters to attach the Quest 3 controllers to the MANUS Universal Mount on the back of the glove. +- Use the HMD's optical hand tracking to position the hands. + +The system will switch dynamically based on the available tracking source. When using controllers it's advised to turn off hand tracking entirely or turn off automatic switching. + ## Troubleshooting -- **Manus SDK not found at build time**: Ensure `ManusSDK` is in `src/plugins/manus/` or `MANUS_SDK_ROOT` is set correctly. -- **Manus SDK not found at runtime**: The build configures RPATH to find the SDK libraries. If you moved the SDK or built without RPATH support, you may need to set `LD_LIBRARY_PATH`. -- **No data available**: Ensure Manus Core is running and gloves are properly connected and calibrated. +- **SDK download fails**: Check your internet connection and try running the install script again +- **Manus SDK not found at build time**: If using manual installation, ensure `ManusSDK` is in `src/plugins/manus/` or `MANUS_SDK_ROOT` is set correctly +- **Manus SDK not found at runtime**: The CMake build configures RPATH to find the SDK libraries. If you moved the SDK, you may need to set `LD_LIBRARY_PATH` +- **No data available**: Ensure Manus Core is running and gloves are properly connected and calibrated +- **CloudXR runtime errors**: Make sure you've sourced `scripts/setup_cloudxr_env.sh` before running the plugin +- **Permission denied for USB devices**: The install script configures udev rules. You may need to run: + ```bash + sudo udevadm control --reload-rules + sudo udevadm trigger + ``` + Then reconnect your Manus devices. ## License diff --git a/src/plugins/manus/core/manus_hand_tracking_plugin.cpp b/src/plugins/manus/core/manus_hand_tracking_plugin.cpp index 9eddb655c..495679f69 100644 --- a/src/plugins/manus/core/manus_hand_tracking_plugin.cpp +++ b/src/plugins/manus/core/manus_hand_tracking_plugin.cpp @@ -1,4 +1,4 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 #include @@ -24,6 +24,30 @@ namespace plugins namespace manus { +namespace +{ + +// Returns true if the OpenXR loader/runtime advertises the given extension. +// xrEnumerateInstanceExtensionProperties is a loader-level function that can be +// called before any XrInstance exists, so this is safe to use at init time. +bool is_openxr_extension_supported(const char* ext_name) +{ + uint32_t count = 0; + if (XR_FAILED(xrEnumerateInstanceExtensionProperties(nullptr, 0, &count, nullptr))) + { + return false; + } + std::vector props(count, XrExtensionProperties{ XR_TYPE_EXTENSION_PROPERTIES }); + if (XR_FAILED(xrEnumerateInstanceExtensionProperties(nullptr, count, &count, props.data()))) + { + return false; + } + return std::any_of(props.begin(), props.end(), + [ext_name](const XrExtensionProperties& p) { return std::string(p.extensionName) == ext_name; }); +} + +} // anonymous namespace + static constexpr XrPosef kLeftHandOffset = { { -0.70710678f, -0.5f, 0.0f, 0.5f }, { -0.1f, 0.02f, -0.02f } }; static constexpr XrPosef kRightHandOffset = { { -0.70710678f, 0.5f, 0.0f, 0.5f }, { 0.1f, 0.02f, -0.02f } }; @@ -38,6 +62,7 @@ void ManusTracker::update() // Update DeviceIOSession which handles time conversion and tracker updates internally if (!m_deviceio_session->update()) { + // Update failed, skip this frame return; } @@ -77,14 +102,14 @@ ManusTracker::~ManusTracker() void ManusTracker::initialize(const std::string& app_name) noexcept(false) { - std::cout << "Initializing Manus SDK..." << std::endl; + std::cout << "[Manus] Initializing SDK..." << std::endl; const SDKReturnCode t_InitializeResult = CoreSdk_InitializeIntegrated(); if (t_InitializeResult != SDKReturnCode::SDKReturnCode_Success) { throw std::runtime_error("Failed to initialize Manus SDK, error code: " + std::to_string(static_cast(t_InitializeResult))); } - std::cout << "Manus SDK initialized successfully" << std::endl; + std::cout << "[Manus] SDK initialized successfully" << std::endl; RegisterCallbacks(); @@ -95,7 +120,7 @@ void ManusTracker::initialize(const std::string& app_name) noexcept(false) t_VUH.view = AxisView::AxisView_ZToViewer; t_VUH.unitScale = 1.0f; - std::cout << "Setting up coordinate system (Z-up, right-handed, meters)..." << std::endl; + std::cout << "[Manus] Setting up coordinate system (Y-up, right-handed, meters)..." << std::endl; const SDKReturnCode t_CoordinateResult = CoreSdk_InitializeCoordinateSystemWithVUH(t_VUH, true); if (t_CoordinateResult != SDKReturnCode::SDKReturnCode_Success) @@ -103,7 +128,7 @@ void ManusTracker::initialize(const std::string& app_name) noexcept(false) throw std::runtime_error("Failed to initialize Manus SDK coordinate system, error code: " + std::to_string(static_cast(t_CoordinateResult))); } - std::cout << "Coordinate system initialized successfully" << std::endl; + std::cout << "[Manus] Coordinate system initialized successfully" << std::endl; ConnectToGloves(); @@ -112,28 +137,70 @@ void ManusTracker::initialize(const std::string& app_name) noexcept(false) try { - // Create ControllerTracker and DeviceIOSession + // Create ControllerTracker unconditionally; HandTracker requires + // XR_EXT_hand_tracking which is optional — only add it when the runtime + // advertises support so xrCreateInstance does not fail with + // XR_ERROR_EXTENSION_NOT_PRESENT on runtimes that lack the extension. m_controller_tracker = std::make_shared(); std::vector> trackers = { m_controller_tracker }; + const bool hand_tracking_supported = is_openxr_extension_supported(XR_EXT_HAND_TRACKING_EXTENSION_NAME); + if (hand_tracking_supported) + { + m_hand_tracker = std::make_shared(); + trackers.push_back(m_hand_tracker); + } + else + { + std::cout << "[Manus] " << XR_EXT_HAND_TRACKING_EXTENSION_NAME + << " is not supported by the current runtime; HandTracker will not be created." << std::endl; + } + // Get required extensions from trackers auto extensions = core::DeviceIOSession::get_required_extensions(trackers); extensions.push_back(XR_NVX1_DEVICE_INTERFACE_BASE_EXTENSION_NAME); + // XR_MNDX_XDEV_SPACE_EXTENSION_NAME is optional: it enables optical (HMD) hand + // tracking as a higher-quality wrist source. If the runtime does not advertise + // it we fall back to controller-based tracking instead of crashing. + const bool xdev_extension_supported = is_openxr_extension_supported(XR_MNDX_XDEV_SPACE_EXTENSION_NAME); + if (xdev_extension_supported) + { + extensions.push_back(XR_MNDX_XDEV_SPACE_EXTENSION_NAME); + } + else + { + std::cout << "[Manus] " << XR_MNDX_XDEV_SPACE_EXTENSION_NAME + << " is not supported by the current runtime; optical hand tracking" + << " will not be available and controller fallback will be used." << std::endl; + } + // Create session with required extensions - constructor automatically begins the session m_session = std::make_shared(app_name, extensions); - auto handles = m_session->get_handles(); + m_handles = m_session->get_handles(); + + // Initialize time converter now that handles are ready + m_time_converter.emplace(m_handles); - // Initialize hand injectors (one per hand) and time converter + // Initialize hand injectors (one per hand) m_left_injector = std::make_unique( - handles.instance, handles.session, XR_HAND_LEFT_EXT, handles.space); + m_handles.instance, m_handles.session, XR_HAND_LEFT_EXT, m_handles.space); m_right_injector = std::make_unique( - handles.instance, handles.session, XR_HAND_RIGHT_EXT, handles.space); - m_time_converter.emplace(handles); + m_handles.instance, m_handles.session, XR_HAND_RIGHT_EXT, m_handles.space); + + m_deviceio_session = core::DeviceIOSession::run(trackers, m_handles); + + // Only attempt XDev hand tracker setup when the extension was actually enabled. + // Skipping here avoids calling xrGetInstanceProcAddr for MNDX entry points that + // the runtime would not have loaded. + if (xdev_extension_supported) + { + initialize_xdev_hand_trackers(); + } - m_deviceio_session = core::DeviceIOSession::run(trackers, handles); + std::cout << "[Manus] Initialized with wrist source: " << (m_xdev_available ? "HandTracking" : "Controllers") + << std::endl; - std::cout << "OpenXR session, HandInjector and DeviceIOSession initialized" << std::endl; success = true; } catch (const std::exception& e) @@ -158,6 +225,9 @@ void ManusTracker::initialize(const std::string& app_name) noexcept(false) void ManusTracker::shutdown_sdk() { + // Cleanup XDev hand trackers first + cleanup_xdev_hand_trackers(); + CoreSdk_RegisterCallbackForRawSkeletonStream(nullptr); CoreSdk_RegisterCallbackForLandscapeStream(nullptr); CoreSdk_RegisterCallbackForErgonomicsStream(nullptr); @@ -319,10 +389,9 @@ void ManusTracker::OnLandscapeStream(const Landscape* landscape) return; } - // Determine which sides are present in this landscape update. + // Extract glove IDs from landscape data bool left_present = false; bool right_present = false; - for (uint32_t i = 0; i < gloves.gloveCount; i++) { const GloveLandscapeData& glove = gloves.gloves[i]; @@ -338,22 +407,250 @@ void ManusTracker::OnLandscapeStream(const Landscape* landscape) } } - // If a glove that was previously known is no longer in the landscape, it has - // disconnected. Clear its cached nodes and reset its injector so readers see - // isActive=false rather than a stale pose. - if (!left_present && tracker.left_glove_id.has_value()) + // Clear stale state for any glove that is no longer present in this landscape + // update (i.e., disconnected). Resetting the IDs prevents OnSkeletonStream from + // matching future packets to a dead glove, and clearing the node cache prevents + // inject_hand_data() from replaying the last known stale pose indefinitely. { - tracker.left_glove_id.reset(); std::lock_guard skeleton_lock(tracker.m_skeleton_mutex); - tracker.m_left_hand_nodes.clear(); + if (!left_present && tracker.left_glove_id.has_value()) + { + std::cout << "[Manus] Left glove disconnected (ID " << *tracker.left_glove_id << ")" << std::endl; + tracker.left_glove_id.reset(); + tracker.m_left_hand_nodes.clear(); + } + if (!right_present && tracker.right_glove_id.has_value()) + { + std::cout << "[Manus] Right glove disconnected (ID " << *tracker.right_glove_id << ")" << std::endl; + tracker.right_glove_id.reset(); + tracker.m_right_hand_nodes.clear(); + } } +} - if (!right_present && tracker.right_glove_id.has_value()) +void ManusTracker::initialize_xdev_hand_trackers() +{ + // Load XDev extension function pointers + auto load_func = [this](const char* name, PFN_xrVoidFunction* ptr) -> bool { - tracker.right_glove_id.reset(); - std::lock_guard skeleton_lock(tracker.m_skeleton_mutex); - tracker.m_right_hand_nodes.clear(); + XrResult result = m_handles.xrGetInstanceProcAddr(m_handles.instance, name, ptr); + return XR_SUCCEEDED(result) && *ptr != nullptr; + }; + + // Load XDev extension functions + if (!load_func("xrCreateXDevListMNDX", reinterpret_cast(&m_pfn_create_xdev_list)) || + !load_func("xrDestroyXDevListMNDX", reinterpret_cast(&m_pfn_destroy_xdev_list)) || + !load_func("xrEnumerateXDevsMNDX", reinterpret_cast(&m_pfn_enumerate_xdevs)) || + !load_func("xrGetXDevPropertiesMNDX", reinterpret_cast(&m_pfn_get_xdev_properties))) + { + std::cerr << "[Manus] XR_MNDX_xdev_space extension not available, falling back to controllers" << std::endl; + return; + } + + // Load hand tracking extension functions + if (!load_func("xrCreateHandTrackerEXT", reinterpret_cast(&m_pfn_create_hand_tracker)) || + !load_func("xrDestroyHandTrackerEXT", reinterpret_cast(&m_pfn_destroy_hand_tracker)) || + !load_func("xrLocateHandJointsEXT", reinterpret_cast(&m_pfn_locate_hand_joints))) + { + std::cerr << "[Manus] Hand tracking extension not available, falling back to controllers" << std::endl; + return; + } + + // Create XDev list + XrCreateXDevListInfoMNDX create_info{ XR_TYPE_CREATE_XDEV_LIST_INFO_MNDX }; + XrResult result = m_pfn_create_xdev_list(m_handles.session, &create_info, &m_xdev_list); + if (XR_FAILED(result)) + { + std::cerr << "[Manus] Failed to create XDevList, falling back to controllers" << std::endl; + return; } + + // Enumerate XDevs + uint32_t xdev_count = 0; + result = m_pfn_enumerate_xdevs(m_xdev_list, 0, &xdev_count, nullptr); + if (XR_FAILED(result) || xdev_count == 0) + { + std::cerr << "[Manus] No XDevs found, falling back to controllers" << std::endl; + return; + } + + std::vector xdev_ids(xdev_count); + result = m_pfn_enumerate_xdevs(m_xdev_list, xdev_count, &xdev_count, xdev_ids.data()); + if (XR_FAILED(result)) + { + return; + } + + // Find native hand tracking devices by matching against their serial strings. + // + // NOTE: The serial values "Head Device (0)" (left) and "Head Device (1)" (right) are + // NOT defined by the XR_MNDX_xdev_space specification. They are an observed runtime- + // specific naming convention (e.g. Monado). If a runtime changes these display names + // across firmware or software updates the match below will silently fail. + // See: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html (XR_MNDX_xdev_space) + XrXDevIdMNDX left_xdev_id = 0; + XrXDevIdMNDX right_xdev_id = 0; + std::vector seen_serials; + + for (const auto& xdev_id : xdev_ids) + { + XrGetXDevInfoMNDX get_info{ XR_TYPE_GET_XDEV_INFO_MNDX }; + get_info.id = xdev_id; + + XrXDevPropertiesMNDX properties{ XR_TYPE_XDEV_PROPERTIES_MNDX }; + result = m_pfn_get_xdev_properties(m_xdev_list, &get_info, &properties); + if (XR_FAILED(result)) + { + continue; + } + + std::string serial_str = properties.serial ? properties.serial : ""; + seen_serials.push_back(serial_str); + + if (serial_str == "Head Device (0)") + { + left_xdev_id = xdev_id; + } + else if (serial_str == "Head Device (1)") + { + right_xdev_id = xdev_id; + } + } + + if (left_xdev_id == 0 || right_xdev_id == 0) + { + std::string serials_list; + for (const auto& s : seen_serials) + { + if (!serials_list.empty()) + serials_list += ", "; + serials_list += '"'; + serials_list += s; + serials_list += '"'; + } + std::cerr << "[Manus] Could not match optical hand-tracking XDevs by serial. " + << "Expected \"Head Device (0)\" (left) and \"Head Device (1)\" (right), " + << "but found: [" << serials_list << "]. " + << "These serial strings are runtime-specific and may have changed." << std::endl; + } + + // Create hand trackers from XDevs + auto create_tracker = [this](XrXDevIdMNDX xdev_id, XrHandEXT hand, XrHandTrackerEXT& out_tracker) -> bool + { + if (xdev_id == 0) + { + return false; + } + + XrCreateHandTrackerXDevMNDX xdev_create_info{ XR_TYPE_CREATE_HAND_TRACKER_XDEV_MNDX }; + xdev_create_info.xdevList = m_xdev_list; + xdev_create_info.id = xdev_id; + + XrHandTrackerCreateInfoEXT create_info{ XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT }; + create_info.next = &xdev_create_info; + create_info.hand = hand; + create_info.handJointSet = XR_HAND_JOINT_SET_DEFAULT_EXT; + + return XR_SUCCEEDED(m_pfn_create_hand_tracker(m_handles.session, &create_info, &out_tracker)); + }; + + bool left_ok = create_tracker(left_xdev_id, XR_HAND_LEFT_EXT, m_native_left_hand_tracker); + bool right_ok = create_tracker(right_xdev_id, XR_HAND_RIGHT_EXT, m_native_right_hand_tracker); + + if (left_ok && right_ok) + { + m_xdev_available = true; + } + else + { + std::cerr << "[Manus] Failed to create native hand trackers, falling back to controllers" << std::endl; + cleanup_xdev_hand_trackers(); + } +} + +void ManusTracker::cleanup_xdev_hand_trackers() +{ + if (m_native_left_hand_tracker != XR_NULL_HANDLE && m_pfn_destroy_hand_tracker) + { + m_pfn_destroy_hand_tracker(m_native_left_hand_tracker); + m_native_left_hand_tracker = XR_NULL_HANDLE; + } + if (m_native_right_hand_tracker != XR_NULL_HANDLE && m_pfn_destroy_hand_tracker) + { + m_pfn_destroy_hand_tracker(m_native_right_hand_tracker); + m_native_right_hand_tracker = XR_NULL_HANDLE; + } + if (m_xdev_list != XR_NULL_HANDLE && m_pfn_destroy_xdev_list) + { + m_pfn_destroy_xdev_list(m_xdev_list); + m_xdev_list = XR_NULL_HANDLE; + } + m_xdev_available = false; +} + +bool ManusTracker::update_xdev_hand(XrHandTrackerEXT tracker, XrTime time, XrPosef& out_wrist_pose, bool& out_is_tracked) +{ + out_is_tracked = false; + + if (tracker == XR_NULL_HANDLE || !m_pfn_locate_hand_joints || time == 0) + { + return false; + } + + XrHandJointsLocateInfoEXT locate_info{ XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT }; + locate_info.baseSpace = m_handles.space; + locate_info.time = time; + + XrHandJointLocationEXT joint_locations[XR_HAND_JOINT_COUNT_EXT]; + + XrHandJointLocationsEXT locations{ XR_TYPE_HAND_JOINT_LOCATIONS_EXT }; + locations.jointCount = XR_HAND_JOINT_COUNT_EXT; + locations.jointLocations = joint_locations; + + XrResult result = m_pfn_locate_hand_joints(tracker, &locate_info, &locations); + if (XR_FAILED(result) || !locations.isActive) + { + return false; + } + + const auto& wrist = joint_locations[XR_HAND_JOINT_WRIST_EXT]; + const bool is_valid = (wrist.locationFlags & XR_SPACE_LOCATION_POSITION_VALID_BIT) && + (wrist.locationFlags & XR_SPACE_LOCATION_ORIENTATION_VALID_BIT); + + if (is_valid) + { + out_wrist_pose = wrist.pose; + // Distinguish actively tracked from valid-but-predicted/stale poses so + // callers can advertise TRACKED bits only when the runtime confirms it. + out_is_tracked = (wrist.locationFlags & XR_SPACE_LOCATION_POSITION_TRACKED_BIT) && + (wrist.locationFlags & XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT); + return true; + } + + return false; +} + +bool ManusTracker::get_controller_wrist_pose(bool is_left, XrPosef& out_wrist_pose) +{ + const auto& tracked = is_left ? m_controller_tracker->get_left_controller(*m_deviceio_session) : + m_controller_tracker->get_right_controller(*m_deviceio_session); + + if (!tracked.data) + { + return false; + } + + bool aim_valid = false; + XrPosef raw_pose = oxr_utils::get_aim_pose(*tracked.data, aim_valid); + + if (!aim_valid) + { + return false; + } + + XrPosef offset_pose = is_left ? kLeftHandOffset : kRightHandOffset; + out_wrist_pose = oxr_utils::multiply_poses(raw_pose, offset_pose); + return true; } void ManusTracker::inject_hand_data() @@ -367,65 +664,63 @@ void ManusTracker::inject_hand_data() right_nodes = m_right_hand_nodes; } - // Get controller data from DeviceIOSession - const auto& left_tracked = m_controller_tracker->get_left_controller(*m_deviceio_session); - const auto& right_tracked = m_controller_tracker->get_right_controller(*m_deviceio_session); - - // Use the OpenXR runtime clock for injection time so it aligns with the - // runtime's own time domain (XrTime), rather than a raw steady_clock cast. + // Get current XrTime from the system monotonic clock XrTime time = m_time_converter->os_monotonic_now(); - auto process_hand = - [&](const std::vector& nodes, bool is_left, std::unique_ptr& injector) + auto process_hand = [&](const std::vector& nodes, bool is_left) { if (nodes.empty()) { - // Glove has disconnected — reset the injector so the runtime sees - // isActive=false. No-op if already null. - injector.reset(); return; } - if (!injector) - { - // Glove reconnected — lazily recreate the push device. - const auto handles = m_session->get_handles(); - XrHandEXT hand = is_left ? XR_HAND_LEFT_EXT : XR_HAND_RIGHT_EXT; - injector = - std::make_unique(handles.instance, handles.session, hand, handles.space); - } - XrHandJointLocationEXT joints[XR_HAND_JOINT_COUNT_EXT]; XrPosef root_pose = { { 0.0f, 0.0f, 0.0f, 1.0f }, { 0.0f, 0.0f, 0.0f } }; bool is_root_tracked = false; - // Get controller snapshot for this hand - const auto& tracked = is_left ? left_tracked : right_tracked; - - if (tracked.data) + // Get wrist pose - auto-select hand tracking or controllers + XrPosef wrist_pose; + bool xdev_pose_valid = false; + if (m_xdev_available) { - bool aim_valid = false; - XrPosef raw_pose = oxr_utils::get_aim_pose(*tracked.data, aim_valid); - - if (aim_valid) + XrHandTrackerEXT tracker = is_left ? m_native_left_hand_tracker : m_native_right_hand_tracker; + bool xdev_tracked = false; + if (update_xdev_hand(tracker, time, wrist_pose, xdev_tracked)) { - XrPosef offset_pose = is_left ? kLeftHandOffset : kRightHandOffset; - XrPosef new_root = oxr_utils::multiply_poses(raw_pose, offset_pose); - + // Cache the pose (valid even when only predicted/stale) so the + // last good pose is available if tracking is briefly interrupted. if (is_left) { - m_left_root_pose = new_root; + m_left_root_pose = wrist_pose; } else { - m_right_root_pose = new_root; + m_right_root_pose = wrist_pose; } - is_root_tracked = true; + // Only mark as tracked when the runtime confirms active tracking; + // a valid-but-untracked pose must not have TRACKED bits set. + is_root_tracked = xdev_tracked; + xdev_pose_valid = true; } } - root_pose = is_left ? m_left_root_pose : m_right_root_pose; + // Fall back to controllers only when xdev provided no valid pose at all. + // If xdev gave a valid-but-untracked pose we keep it rather than + // overwriting it with a controller pose that would be falsely marked tracked. + if (!xdev_pose_valid && get_controller_wrist_pose(is_left, wrist_pose)) + { + if (is_left) + { + m_left_root_pose = wrist_pose; + } + else + { + m_right_root_pose = wrist_pose; + } + is_root_tracked = true; + } + root_pose = is_left ? m_left_root_pose : m_right_root_pose; uint32_t nodes_count = static_cast(nodes.size()); for (uint32_t j = 0; j < XR_HAND_JOINT_COUNT_EXT; j++) @@ -486,18 +781,16 @@ void ManusTracker::inject_hand_data() if (is_left) { - if (m_left_injector) - m_left_injector->push(joints, time); + m_left_injector->push(joints, time); } else { - if (m_right_injector) - m_right_injector->push(joints, time); + m_right_injector->push(joints, time); } }; - process_hand(left_nodes, true, m_left_injector); - process_hand(right_nodes, false, m_right_injector); + process_hand(left_nodes, true); + process_hand(right_nodes, false); } } // namespace manus diff --git a/src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp b/src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp index 5b10b0d64..325d3a8be 100644 --- a/src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp +++ b/src/plugins/manus/inc/core/manus_hand_tracking_plugin.hpp @@ -5,11 +5,14 @@ #include #include +#include #include +#include #include #include #include +#include #include #include #include @@ -29,6 +32,7 @@ namespace manus class __attribute__((visibility("default"))) ManusTracker { public: + /// Get the singleton instance static ManusTracker& instance(const std::string& app_name = "ManusHandPlugin") noexcept(false); void update(); @@ -56,6 +60,14 @@ class __attribute__((visibility("default"))) ManusTracker // OpenXR specific methods void inject_hand_data(); + void initialize_xdev_hand_trackers(); + void cleanup_xdev_hand_trackers(); + // Returns true if a valid (POSITION_VALID | ORIENTATION_VALID) wrist pose was + // obtained. out_is_tracked is set to true only when the runtime also reports + // POSITION_TRACKED | ORIENTATION_TRACKED, meaning the pose is actively tracked + // rather than predicted/stale. + bool update_xdev_hand(XrHandTrackerEXT tracker, XrTime time, XrPosef& out_wrist_pose, bool& out_is_tracked); + bool get_controller_wrist_pose(bool is_left, XrPosef& out_wrist_pose); // -- Member Variables -- @@ -71,11 +83,28 @@ class __attribute__((visibility("default"))) ManusTracker // OpenXR State std::shared_ptr m_session; + core::OpenXRSessionHandles m_handles; std::unique_ptr m_left_injector; std::unique_ptr m_right_injector; - std::optional m_time_converter; std::shared_ptr m_controller_tracker; + std::shared_ptr m_hand_tracker; std::unique_ptr m_deviceio_session; + + // XDev native hand trackers (Quest 3 hand tracking via XR_MNDX_xdev_space) + XrXDevListMNDX m_xdev_list = XR_NULL_HANDLE; + XrHandTrackerEXT m_native_left_hand_tracker = XR_NULL_HANDLE; + XrHandTrackerEXT m_native_right_hand_tracker = XR_NULL_HANDLE; + bool m_xdev_available = false; + + // XDev function pointers + PFN_xrCreateXDevListMNDX m_pfn_create_xdev_list = nullptr; + PFN_xrDestroyXDevListMNDX m_pfn_destroy_xdev_list = nullptr; + PFN_xrEnumerateXDevsMNDX m_pfn_enumerate_xdevs = nullptr; + PFN_xrGetXDevPropertiesMNDX m_pfn_get_xdev_properties = nullptr; + PFN_xrCreateHandTrackerEXT m_pfn_create_hand_tracker = nullptr; + PFN_xrDestroyHandTrackerEXT m_pfn_destroy_hand_tracker = nullptr; + PFN_xrLocateHandJointsEXT m_pfn_locate_hand_joints = nullptr; + // Persistent root poses (initialized to identity) XrPosef m_left_root_pose = { { 0.0f, 0.0f, 0.0f, 1.0f }, { 0.0f, 0.0f, 0.0f } }; XrPosef m_right_root_pose = { { 0.0f, 0.0f, 0.0f, 1.0f }, { 0.0f, 0.0f, 0.0f } }; @@ -84,6 +113,9 @@ class __attribute__((visibility("default"))) ManusTracker mutable std::mutex m_skeleton_mutex; std::vector m_left_hand_nodes; std::vector m_right_hand_nodes; + + // Time converter for XR timestamps (initialized after handles are ready) + std::optional m_time_converter; }; } // namespace manus diff --git a/src/plugins/manus/install-dependencies.sh b/src/plugins/manus/install-dependencies.sh new file mode 100644 index 000000000..40f4fecda --- /dev/null +++ b/src/plugins/manus/install-dependencies.sh @@ -0,0 +1,191 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -e # Exit on error +set -u # Exit on undefined variable + +echo "=== Dependency Setup for Debian/Ubuntu ===" + +# Install required packages including GCC-11 for compatibility +echo "[1/7] Installing build dependencies and GCC-11..." + +# Check if GCC-11 is available, if not add appropriate repo +if ! apt-cache show gcc-11 &>/dev/null; then + echo "GCC-11 not found in current repos." + # Detect if running Ubuntu or Debian + if grep -qi "ubuntu" /etc/os-release; then + echo "Adding Ubuntu Toolchain PPA for GCC-11..." + apt-get install -y software-properties-common + add-apt-repository -y ppa:ubuntu-toolchain-r/test + else + echo "Adding Debian 12 (Bookworm) repository for GCC-11..." + apt-get install -y debian-archive-keyring + echo "deb http://deb.debian.org/debian bookworm main" | tee /etc/apt/sources.list.d/bookworm-gcc.list + + # Pin only the GCC-11 packages to Bookworm; everything else stays at the + # default priority (500) so unrelated Bookworm packages are never chosen + # over the host distro's packages. + cat > /etc/apt/preferences.d/bookworm-gcc.pref << 'EOF' +# Allow only gcc-11 / g++-11 / libgcc-11 to be installed from Bookworm. +# All other packages from this release keep the default low priority (1) +# so they cannot shadow packages from the host distribution. +Package: * +Pin: release n=bookworm +Pin-Priority: 1 + +Package: gcc-11 g++-11 libgcc-11-dev libgcc-s1 cpp-11 +Pin: release n=bookworm +Pin-Priority: 900 +EOF + fi +fi + +apt-get update +apt-get install -y \ + build-essential \ + gcc-11 \ + g++-11 \ + cmake \ + git \ + libtool \ + libssl-dev \ + zlib1g-dev + +echo "Using GCC-11 to compile gRPC v1.28.1 (GCC-12+ is incompatible)" + +echo "[2/7] Cloning gRPC repository..." +mkdir -p /var/local/git +if [ ! -d "/var/local/git/grpc" ]; then + git clone --recurse-submodules -b v1.28.1 --depth 1 https://github.com/grpc/grpc /var/local/git/grpc +else + # Normalize the existing checkout so the build always operates on exactly + # v1.28.1, regardless of any prior modifications or partial updates. + echo "gRPC repository already exists; resetting to v1.28.1..." + cd /var/local/git/grpc + # Unshallow if this is a shallow clone so we can fetch/checkout freely + if [ -f .git/shallow ]; then + git fetch --unshallow origin + fi + git fetch origin refs/tags/v1.28.1:refs/tags/v1.28.1 --force + git checkout v1.28.1 + git reset --hard v1.28.1 + git clean -ffdx + git submodule sync --recursive + git submodule update --init --recursive + cd /var/local/git +fi + +# Check if running on x86_64 +if [ "$(uname -m)" == "x86_64" ]; then + echo "[3/7] Building protobuf for x86_64..." && \ + cd /var/local/git/grpc/third_party/protobuf && \ + ./autogen.sh && ./configure --enable-shared && \ + make -j$(nproc) && make -j$(nproc) check && make install && ldconfig +else + echo "[3/7] Not running on x86_64, Building protobuf for x86_64 SKIPPED." +fi + +# Patch abseil-cpp for GCC 11+ compatibility +echo "[4/7] Patching abseil-cpp for GCC compatibility..." +if ! grep -q '#include ' /var/local/git/grpc/third_party/abseil-cpp/absl/synchronization/internal/graphcycles.cc; then + sed -i '1i#include ' /var/local/git/grpc/third_party/abseil-cpp/absl/synchronization/internal/graphcycles.cc +fi +if grep -q 'std::max(SIGSTKSZ, 65536)' /var/local/git/grpc/third_party/abseil-cpp/absl/debugging/failure_signal_handler.cc 2>/dev/null; then + sed -i 's/std::max(SIGSTKSZ, 65536)/std::max(SIGSTKSZ, 65536)/' /var/local/git/grpc/third_party/abseil-cpp/absl/debugging/failure_signal_handler.cc +fi + +# Add missing cstdint include for uint64_t +if ! grep -q '#include ' /var/local/git/grpc/third_party/abseil-cpp/absl/strings/internal/str_format/extension.h; then + sed -i '/#include "absl\/strings\/string_view.h"/a #include ' /var/local/git/grpc/third_party/abseil-cpp/absl/strings/internal/str_format/extension.h + echo "Added include to extension.h" +fi + +# Build protobuf natively with GCC-11 +if [ "$(uname -m)" == "aarch64" ]; then + echo "[5/7] Building protobuf for ARM64 with GCC-11..." + mkdir -p /var/local/git/grpc/third_party/protobuf/build + cd /var/local/git/grpc/third_party/protobuf/build + CC=gcc-11 CXX=g++-11 cmake ../cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=/usr/local \ + -Dprotobuf_BUILD_TESTS=OFF \ + -Dprotobuf_WITH_ZLIB=ON \ + -Dprotobuf_BUILD_SHARED_LIBS=ON + make -j$(nproc) + make install + ldconfig +else + echo "[5/7] Not running on aarch64, Building protobuf for ARM64 SKIPPED." +fi + +echo "Protobuf installed: $(protoc --version)" + +# Build gRPC natively with GCC-11 +echo "[6/7] Building gRPC v1.28.1 with GCC-11..." +mkdir -p /var/local/git/grpc/build +cd /var/local/git/grpc/build +CC=gcc-11 CXX=g++-11 cmake .. \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=/usr/local \ + -DgRPC_INSTALL=ON \ + -DgRPC_BUILD_TESTS=OFF \ + -DBUILD_SHARED_LIBS=ON \ + -DgRPC_PROTOBUF_PROVIDER=package \ + -DgRPC_ZLIB_PROVIDER=package \ + -DgRPC_CARES_PROVIDER=module \ + -DgRPC_SSL_PROVIDER=package \ + -DgRPC_BUILD_CSHARP_EXT=OFF +make -j$(nproc) +make install +ldconfig + +echo "[7/7] Installing additional dependencies..." +apt-get install -y \ + libc-ares-dev \ + libzmq3-dev \ + libncurses-dev \ + libudev-dev \ + libusb-1.0-0-dev + +# Configure library path persistently +echo "[Setup] Configuring library path..." +if [ ! -f "/etc/ld.so.conf.d/local-libs.conf" ]; then + echo "/usr/local/lib" | tee /etc/ld.so.conf.d/local-libs.conf > /dev/null + ldconfig + echo "Added /usr/local/lib to ld.so.conf" +fi + +# Add read/write permissions for manus devices +if [ ! -f "/etc/udev/rules.d/70-manus-hid.rules" ]; then + mkdir -p /etc/udev/rules.d/ + touch /etc/udev/rules.d/70-manus-hid.rules + echo "[Setup] Adding read/write permissions for manus devices..." + echo "# HIDAPI/libusb" >> /etc/udev/rules.d/70-manus-hid.rules + echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"3325\", MODE:=\"0666\"" >> /etc/udev/rules.d/70-manus-hid.rules + echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"1915\", ATTRS{idProduct}==\"83fd\", MODE:=\"0666\"" >> /etc/udev/rules.d/70-manus-hid.rules + echo "# HIDAPI/hidraw" >> /etc/udev/rules.d/70-manus-hid.rules + echo "KERNEL==\"hidraw*\", ATTRS{idVendor}==\"3325\", MODE:=\"0666\"" >> /etc/udev/rules.d/70-manus-hid.rules +fi + +echo "" +echo "=== Setup Complete ===" +echo "gRPC v1.28.1 and protobuf compiled successfully with GCC-11" +echo "Libraries installed to: /usr/local/lib" +echo "Headers installed to: /usr/local/include" +echo "" +echo "To use these libraries in your current shell session:" +echo " export LD_LIBRARY_PATH=/usr/local/lib" +echo "" +echo "Library path has been configured system-wide via ldconfig." +echo "" +echo "Udev rules have been configured to allow read/write access to manus devices." +echo "You may need to reload udev rules for the changes to take effect by running the following commands:" +echo " sudo udevadm control --reload-rules" +echo " sudo udevadm trigger" +echo "" +echo "Installed versions:" +echo " - Protobuf: $(protoc --version)" +echo " - Compiler: $(gcc-11 --version | head -n1)" +echo "" +echo "You can now build your SDK project with gRPC v1.28.1." \ No newline at end of file diff --git a/src/plugins/manus/install_manus.sh b/src/plugins/manus/install_manus.sh new file mode 100755 index 000000000..d2eedea95 --- /dev/null +++ b/src/plugins/manus/install_manus.sh @@ -0,0 +1,209 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -e # Exit on error +set -u # Exit on undefined variable + +echo "=== MANUS SDK Installation Script ===" +echo "" + +# Define SDK download URL and version +MANUS_SDK_VERSION="3.1.1" +MANUS_SDK_URL="https://static.manus-meta.com/resources/manus_core_3/sdk/MANUS_Core_${MANUS_SDK_VERSION}_SDK.zip" +MANUS_SDK_ZIP="MANUS_Core_${MANUS_SDK_VERSION}_SDK.zip" +MANUS_SDK_SHA256="c5ccd3c42a501107ec79f70d8450a486fbc3925c5c1e18e606114d09f2d9d24a" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Detect architecture +ARCH=$(uname -m) +echo "Detected architecture: $ARCH" +echo "" + +# Query user for dependency installation type +echo "MANUS Core dependencies installation options:" +echo " 1) Install MANUS Core Integrated dependencies only (faster)" +echo " 2) Install both MANUS Core Integrated and Remote dependencies (includes gRPC, takes longer)" +echo "" +read -p "Enter your choice (1 or 2): " INSTALL_CHOICE + +case "$INSTALL_CHOICE" in + 1) + echo "Installing MANUS Core Integrated dependencies only..." + INSTALL_REMOTE=false + ;; + 2) + echo "Installing MANUS Core Integrated and Remote dependencies..." + INSTALL_REMOTE=true + ;; + *) + echo "Invalid choice. Defaulting to option 1 (Integrated only)." + INSTALL_REMOTE=false + ;; +esac +echo "" + +# Run the dependency installation script +if [ -f "$SCRIPT_DIR/install-dependencies.sh" ]; then + echo "[1/4] Running dependency installation script..." + if [ "$INSTALL_REMOTE" = true ]; then + sudo bash "$SCRIPT_DIR/install-dependencies.sh" + else + echo "Skipping Remote dependencies (gRPC installation)..." + echo "Installing minimal dependencies only..." + sudo apt-get update + sudo apt-get install -y \ + build-essential \ + cmake \ + curl \ + git \ + libssl-dev \ + unzip \ + zlib1g-dev \ + libc-ares-dev \ + libzmq3-dev \ + libncurses-dev \ + libudev-dev \ + libusb-1.0-0-dev + + # Add read/write permissions for manus devices + if [ ! -f "/etc/udev/rules.d/70-manus-hid.rules" ]; then + sudo mkdir -p /etc/udev/rules.d/ + sudo touch /etc/udev/rules.d/70-manus-hid.rules + echo "Adding read/write permissions for manus devices..." + echo "# HIDAPI/libusb" | sudo tee -a /etc/udev/rules.d/70-manus-hid.rules > /dev/null + echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"3325\", MODE:=\"0666\"" | sudo tee -a /etc/udev/rules.d/70-manus-hid.rules > /dev/null + echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"1915\", ATTRS{idProduct}==\"83fd\", MODE:=\"0666\"" | sudo tee -a /etc/udev/rules.d/70-manus-hid.rules > /dev/null + echo "# HIDAPI/hidraw" | sudo tee -a /etc/udev/rules.d/70-manus-hid.rules > /dev/null + echo "KERNEL==\"hidraw*\", ATTRS{idVendor}==\"3325\", MODE:=\"0666\"" | sudo tee -a /etc/udev/rules.d/70-manus-hid.rules > /dev/null + sudo udevadm control --reload-rules + fi + fi + echo "" +else + echo "Warning: install-dependencies.sh not found. Skipping dependency installation." + echo "" +fi + +# Download MANUS SDK +echo "[2/4] Downloading MANUS SDK v${MANUS_SDK_VERSION}..." +cd "$SCRIPT_DIR" + +if [ -f "$MANUS_SDK_ZIP" ]; then + echo "SDK archive already exists. Skipping download." +else + if command -v curl &> /dev/null; then + # -f: fail on HTTP errors (4xx/5xx), -L: follow redirects + curl -fL "$MANUS_SDK_URL" -o "$MANUS_SDK_ZIP" + elif command -v wget &> /dev/null; then + # --server-response prints HTTP status; wget already exits non-zero on errors + wget --server-response "$MANUS_SDK_URL" -O "$MANUS_SDK_ZIP" + else + echo "Error: Neither curl nor wget found. Please install curl (apt-get install curl)." + exit 1 + fi +fi + +# Verify archive integrity before extracting +if [ -n "${MANUS_SDK_SHA256:-}" ]; then + echo "Verifying SDK archive checksum..." + ACTUAL_SHA256=$(sha256sum "$MANUS_SDK_ZIP" | awk '{print $1}') + if [ "$ACTUAL_SHA256" != "$MANUS_SDK_SHA256" ]; then + echo "Error: SHA-256 checksum mismatch for $MANUS_SDK_ZIP" + echo " Expected: $MANUS_SDK_SHA256" + echo " Actual: $ACTUAL_SHA256" + echo "The archive may be corrupted or tampered with. Aborting." + rm -f "$MANUS_SDK_ZIP" + exit 1 + fi + echo "Checksum verified." +else + echo "Warning: MANUS_SDK_SHA256 is not set. Skipping checksum verification." + echo " Set MANUS_SDK_SHA256 in install_manus.sh to enable integrity checking." +fi +echo "" + +# Extract SDK and copy ManusSDK folder +echo "[3/4] Extracting MANUS SDK..." + +# Remove existing ManusSDK if present +if [ -d "$SCRIPT_DIR/ManusSDK" ]; then + echo "Removing existing ManusSDK directory..." + rm -rf "$SCRIPT_DIR/ManusSDK" +fi + +# Extract the archive +if command -v unzip &> /dev/null; then + unzip -q "$MANUS_SDK_ZIP" +else + echo "Error: unzip command not found. Please install unzip." + exit 1 +fi + +SDK_CLIENT_DIR="SDKClient_Linux" + +# Find and copy ManusSDK folder +EXTRACTED_DIR=$(find . -maxdepth 1 -type d -name "ManusSDK_v*" | head -n 1) +if [ -z "$EXTRACTED_DIR" ]; then + echo "Error: Could not find extracted SDK directory." + exit 1 +fi + +if [ -d "$EXTRACTED_DIR/$SDK_CLIENT_DIR/ManusSDK" ]; then + echo "Copying ManusSDK folder from $SDK_CLIENT_DIR..." + cp -r "$EXTRACTED_DIR/$SDK_CLIENT_DIR/ManusSDK" "$SCRIPT_DIR/" + echo "ManusSDK copied successfully to $SCRIPT_DIR/ManusSDK" + + if [ "$INSTALL_REMOTE" = true ]; then + echo "Note: Both libManusSDK.so and libManusSDK_Integrated.so available." + echo "CMake will use libManusSDK_Integrated.so by default." + else + echo "Note: Using libManusSDK_Integrated.so (CMake auto-selects)." + fi +else + echo "Error: ManusSDK folder not found in $EXTRACTED_DIR/$SDK_CLIENT_DIR" + exit 1 +fi + +# Clean up extracted directory and zip file +echo "Cleaning up temporary files..." +rm -rf "$EXTRACTED_DIR" +rm -f "$MANUS_SDK_ZIP" +echo "" + +# Build the plugin +echo "[4/4] Building Manus plugin from TeleopCore root..." + +# Find TeleopCore root (3 levels up from src/plugins/manus) +TELEOP_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" +cd "$TELEOP_ROOT" + +echo "TeleopCore root: $TELEOP_ROOT" + +echo "Configuring CMake..." +cmake -S . -B build -DCMAKE_BUILD_TYPE=Release + +# Build the plugin and the diagnostic printer tool +echo "Building..." +cmake --build build --target manus_hand_plugin manus_hand_tracker_printer -j$(nproc) + +# Install only the manus component +echo "Installing..." +cmake --install build --component manus + +echo "" +echo "=== Installation Complete ===" +echo "MANUS SDK v${MANUS_SDK_VERSION} installed to: $SCRIPT_DIR/ManusSDK" +echo "Plugin built and installed successfully" +echo "Plugin executable: $TELEOP_ROOT/install/plugins/manus/manus_hand_plugin" +echo "Printer diagnostic: $TELEOP_ROOT/build/bin/manus_hand_tracker_printer" +echo "" +if [ "$INSTALL_REMOTE" = false ]; then + echo "Note: Only MANUS Core Integrated dependencies were installed." + echo "If you need Remote functionality (gRPC), re-run and select option 2." +fi +echo "" +echo "To reload udev rules (if not already done), run:" +echo " sudo udevadm control --reload-rules" +echo " sudo udevadm trigger" +echo "" diff --git a/src/plugins/manus/tools/manus_hand_tracker_printer.cpp b/src/plugins/manus/tools/manus_hand_tracker_printer.cpp index a4d1cf8a3..5a18a38b8 100644 --- a/src/plugins/manus/tools/manus_hand_tracker_printer.cpp +++ b/src/plugins/manus/tools/manus_hand_tracker_printer.cpp @@ -16,12 +16,12 @@ try (void)argc; (void)argv; - std::cout << "Initializing Manus Tracker..." << std::endl; + std::cout << "[Manus] Initializing Manus Tracker..." << std::endl; // Initialize the Manus tracker auto& tracker = plugins::manus::ManusTracker::instance("ManusHandPrinter"); - std::cout << "Press Ctrl+C to stop. Printing joint data..." << std::endl; + std::cout << "[Manus] Press Ctrl+C to stop. Printing joint data..." << std::endl; int frame = 0; while (true) @@ -32,12 +32,12 @@ try if (left_nodes.empty() && right_nodes.empty()) { - std::cout << "No data available yet..." << std::endl; + std::cout << "[Manus] No data available yet..." << std::endl; std::this_thread::sleep_for(std::chrono::milliseconds(200)); continue; } - std::cout << "\n=== Frame " << frame << " ===" << std::endl; + std::cout << "\n[Manus] === Frame " << frame << " ===" << std::endl; // Helper lambda to print hand data auto print_hand = [](const std::string& side, const std::vector& nodes) @@ -47,14 +47,14 @@ try return; } - std::cout << "\n" << side << " hand (" << nodes.size() << " joints):" << std::endl; + std::cout << "[Manus] " << side << " hand (" << nodes.size() << " joints):" << std::endl; for (size_t i = 0; i < std::min(nodes.size(), static_cast(5)); ++i) { const auto& pos = nodes[i].transform.position; const auto& ori = nodes[i].transform.rotation; - std::cout << " Joint " << i << ": " + std::cout << "[Manus] Joint " << i << ": " << "pos=[" << std::fixed << std::setprecision(3) << pos.x << ", " << pos.y << ", " << pos.z << "] " << "ori=[" << ori.x << ", " << ori.y << ", " << ori.z << ", " << ori.w << "]" << std::endl; @@ -62,7 +62,7 @@ try if (nodes.size() > 5) { - std::cout << " ... (" << (nodes.size() - 5) << " more joints)" << std::endl; + std::cout << "[Manus] ... (" << (nodes.size() - 5) << " more joints)" << std::endl; } };