diff --git a/dimos/teleop/README.md b/dimos/teleop/README.md index e64a7b43ea..fac35ab512 100644 --- a/dimos/teleop/README.md +++ b/dimos/teleop/README.md @@ -1,29 +1,33 @@ # Teleop Stack -Teleoperation modules for DimOS. Currently supports Meta Quest 3 VR controllers. +Teleoperation modules for DimOS. Supports Meta Quest 3 VR controllers and phone motion sensors. ## Architecture ``` -Quest Browser (WebXR) +Quest/Phone Browser │ - │ PoseStamped + Joy via WebSocket + │ LCM-encoded binary via WebSocket ▼ -Deno Bridge (teleop_server.ts) +Embedded FastAPI Server (HTTPS) │ - │ LCM topics + │ Fingerprint-based message dispatch ▼ -QuestTeleopModule - │ WebXR → robot frame transform - │ Pose computation + button state packing +TeleopModule (Quest or Phone) + │ Frame transforms + pose/twist computation ▼ PoseStamped / TwistStamped / Buttons outputs ``` +Each teleop module embeds a `RobotWebInterface` (FastAPI + uvicorn) that: +- Serves the teleop web app at `/teleop` +- Accepts WebSocket connections at `/ws` +- Handles SSL certificate generation for HTTPS (required by mobile sensor APIs) + ## Modules ### QuestTeleopModule -Base teleop module. Gets controller data, computes output poses, and publishes them. Default engage: hold primary button (X/A). Subclass to customize. +Base Quest teleop module. Gets controller data via WebSocket, computes output poses, and publishes them. Default engage: hold primary button (X/A). Subclass to customize. ### ArmTeleopModule Toggle-based engage — press primary button once to engage, press again to disengage. @@ -34,6 +38,12 @@ Outputs TwistStamped (linear + angular velocity) instead of PoseStamped. ### VisualizingTeleopModule Adds Rerun visualization for debugging. Extends ArmTeleopModule (toggle engage). +### PhoneTeleopModule +Base phone teleop module. Receives orientation + gyro data from phone motion sensors, computes velocity commands from orientation deltas. + +### SimplePhoneTeleop +Filters to mobile-base axes (linear.x, linear.y, angular.z) and publishes as `Twist`. + ## Subclassing `QuestTeleopModule` is designed for extension. Override these methods: @@ -60,16 +70,14 @@ teleop/ │ ├── quest_teleop_module.py # Base Quest teleop module │ ├── quest_extensions.py # ArmTeleop, TwistTeleop, VisualizingTeleop │ ├── quest_types.py # QuestControllerState, Buttons -│ └── web/ # Deno bridge + WebXR client -│ ├── teleop_server.ts -│ └── static/index.html +│ └── web/ +│ └── static/index.html # WebXR client ├── phone/ │ ├── phone_teleop_module.py # Base Phone teleop module │ ├── phone_extensions.py # SimplePhoneTeleop │ ├── blueprints.py # Pre-wired configurations -│ └── web/ # Deno bridge + mobile web app -│ ├── teleop_server.ts -│ └── static/index.html +│ └── web/ +│ └── static/index.html # Mobile sensor web app ├── utils/ │ ├── teleop_transforms.py # WebXR → robot frame math │ └── teleop_visualization.py # Rerun visualization helpers @@ -78,4 +86,11 @@ teleop/ ## Quick Start -See [Quest Web README](quest/web/README.md) for running the Deno bridge and connecting the Quest headset. +```bash +dimos run arm-teleop # Quest arm teleop +dimos run phone-go2-teleop # Phone → Go2 +``` + +Open `https://:/teleop` on device. Accept the self-signed certificate. +- Quest: port 8443 +- Phone: port 8444 diff --git a/dimos/teleop/phone/README.md b/dimos/teleop/phone/README.md index dd2af02281..5f8541f602 100644 --- a/dimos/teleop/phone/README.md +++ b/dimos/teleop/phone/README.md @@ -5,66 +5,34 @@ Teleoperation via smartphone motion sensors. Tilt to drive. ## Architecture ``` -Phone Browser (DeviceOrientation + DeviceMotion) - | - | TwistStamped + Bool via WebSocket - v -Deno Bridge (teleop_server.ts) - | - | LCM topics - v -PhoneTeleopModule - | Orientation delta from home pose - | Gains -> velocity commands - v -TwistStamped / Twist outputs +Phone Browser ──WebSocket──→ Embedded HTTPS Server ──→ PhoneTeleopModule +(sensors + button) (port 8444) (delta → velocity) ``` -## Modules +## Running -### PhoneTeleopModule -Base module. Receives raw sensor data and button state. On engage (button hold), captures home orientation and publishes deltas as TwistStamped. Launches the Deno bridge server automatically. +```bash +dimos run phone-go2-teleop # Go2 +dimos run simple-phone-teleop # Generic ground robot +``` -### SimplePhoneTeleop -Filters to mobile-base axes (linear.x, linear.y, angular.z) and publishes as `Twist` on `cmd_vel` for direct autoconnect wiring with any module that has `cmd_vel: In[Twist]`. +Open `https://:8444/teleop` on phone. Accept cert, allow sensors, connect, hold to drive. ## Subclassing -Override these methods: - | Method | Purpose | |--------|---------| | `_handle_engage()` | Customize engage/disengage logic | -| `_should_publish()` | Add conditions for when to publish | | `_publish_msg()` | Change output format | -**Do not acquire `self._lock` in overrides.** The control loop already holds it. - -## LCM Topics - -| Topic | Type | Description | -|-------|------|-------------| -| `/phone_sensors` | TwistStamped | linear=(roll,pitch,yaw) deg, angular=(gyro) deg/s | -| `/phone_button` | Bool | Teleop engage button (1=held) | -| `/teleop/twist` | TwistStamped | Output velocity command | - -## Running - -```bash -dimos run phone-go2-teleop # Go2 -dimos run simple-phone-teleop # Generic ground robot -``` - -Server starts on port `8444`. Open `https://:8444` on phone, accept the self-signed certificate, allow sensor permissions, connect, hold button to drive. +`self._lock` is already held — don't acquire it in overrides. ## File Structure ``` phone/ -├── phone_teleop_module.py # Base phone teleop module +├── phone_teleop_module.py # Base module ├── phone_extensions.py # SimplePhoneTeleop -├── blueprints.py # Pre-wired configurations -└── web/ - ├── teleop_server.ts # Deno WSS-to-LCM bridge - └── static/index.html # Mobile web app +├── blueprints.py +└── web/static/index.html # Mobile web app ``` diff --git a/dimos/teleop/phone/phone_teleop_module.py b/dimos/teleop/phone/phone_teleop_module.py index ec33df58eb..4d40b995f3 100644 --- a/dimos/teleop/phone/phone_teleop_module.py +++ b/dimos/teleop/phone/phone_teleop_module.py @@ -17,43 +17,48 @@ Phone Teleoperation Module. Receives raw sensor data (TwistStamped) and button state (Bool) from the -phone web app via the Deno LCM bridge. Computes orientation deltas from -a initial orientation captured on engage, converts to TwistStamped velocity -commands via configurable gains, and publishes. - +phone web app via an embedded FastAPI WebSocket server. Computes orientation +deltas from an initial orientation captured on engage, converts to TwistStamped +velocity commands via configurable gains, and publishes. """ from dataclasses import dataclass from pathlib import Path -import shutil -import signal -import subprocess import threading import time from typing import Any -from reactivex.disposable import Disposable +from dimos_lcm.geometry_msgs import TwistStamped as LCMTwistStamped +from dimos_lcm.std_msgs import Bool as LCMBool +from fastapi import WebSocket, WebSocketDisconnect +from fastapi.responses import HTMLResponse +from fastapi.staticfiles import StaticFiles from dimos.core.core import rpc from dimos.core.module import Module, ModuleConfig -from dimos.core.stream import In, Out +from dimos.core.stream import Out from dimos.msgs.geometry_msgs import Twist, TwistStamped, Vector3 from dimos.msgs.std_msgs.Bool import Bool from dimos.utils.logging_config import setup_logger +from dimos.utils.path_utils import get_project_root +from dimos.web.robot_web_interface import RobotWebInterface logger = setup_logger() +STATIC_DIR = Path(__file__).parent / "web" / "static" + @dataclass class PhoneTeleopConfig(ModuleConfig): control_loop_hz: float = 50.0 linear_gain: float = 1.0 / 30.0 # Gain: maps degrees of tilt to m/s. 30 deg -> 1.0 m/s angular_gain: float = 1.0 / 30.0 # Gain: maps gyro deg/s to rad/s. 30 deg/s -> 1.0 rad/s + server_port: int = 8444 class PhoneTeleopModule(Module[PhoneTeleopConfig]): """ - Receives raw sensor data from the phone web app: + Receives raw sensor data from the phone web app via an embedded WebSocket server: - TwistStamped: linear=(roll, pitch, yaw) deg, angular=(gyro) deg/s - Bool: teleop button state (True = held) @@ -63,9 +68,6 @@ class PhoneTeleopModule(Module[PhoneTeleopConfig]): default_config = PhoneTeleopConfig - # Inputs from Deno bridge - phone_sensors: In[TwistStamped] - phone_button: In[Bool] # Output: velocity command to robot twist_output: Out[TwistStamped] @@ -86,9 +88,52 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: self._control_loop_thread: threading.Thread | None = None self._stop_event = threading.Event() - # Deno bridge server - self._server_process: subprocess.Popen[bytes] | None = None - self._server_script = Path(__file__).parent / "web" / "teleop_server.ts" + # Embedded web server — RobotWebInterface provides FastAPI app + run()/shutdown() + self._web_server = RobotWebInterface(port=self.config.server_port) + self._web_server_thread: threading.Thread | None = None + + # Fingerprint-based message dispatch table + self._decoders: dict[bytes, Any] = { + LCMTwistStamped._get_packed_fingerprint(): self._on_sensors_bytes, + LCMBool._get_packed_fingerprint(): self._on_button_bytes, + } + + self._setup_routes() + + # ------------------------------------------------------------------------- + # Web Server Routes + # ------------------------------------------------------------------------- + + def _setup_routes(self) -> None: + """Register teleop routes on the embedded web server.""" + + @self._web_server.app.get("/teleop", response_class=HTMLResponse) + async def teleop_index() -> HTMLResponse: + index_path = STATIC_DIR / "index.html" + return HTMLResponse(content=index_path.read_text()) + + if STATIC_DIR.is_dir(): + self._web_server.app.mount( + "/static", StaticFiles(directory=str(STATIC_DIR)), name="teleop_static" + ) + + @self._web_server.app.websocket("/ws") + async def websocket_endpoint(ws: WebSocket) -> None: + await ws.accept() + logger.info("Phone client connected") + try: + while True: + data = await ws.receive_bytes() + fingerprint = data[:8] + decoder = self._decoders.get(fingerprint) + if decoder: + decoder(data) + else: + logger.warning(f"Unknown message fingerprint: {fingerprint.hex()}") + except WebSocketDisconnect: + logger.info("Phone client disconnected") + except Exception: + logger.exception("WebSocket error") # ------------------------------------------------------------------------- # Lifecycle @@ -97,11 +142,6 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: @rpc def start(self) -> None: super().start() - for stream, handler in ( - (self.phone_sensors, self._on_sensors), - (self.phone_button, self._on_button), - ): - self._disposables.add(Disposable(stream.subscribe(handler))) # type: ignore[attr-defined] self._start_server() self._start_control_loop() @@ -132,73 +172,47 @@ def _disengage(self) -> None: logger.info("Phone teleop disengaged") # ------------------------------------------------------------------------- - # Callbacks + # WebSocket Message Decoders # ------------------------------------------------------------------------- - def _on_sensors(self, msg: TwistStamped) -> None: - """Callback for raw sensor TwistStamped from the phone""" + def _on_sensors_bytes(self, data: bytes) -> None: + """Decode raw LCM bytes into TwistStamped and update sensor state.""" + msg = TwistStamped.lcm_decode(data) with self._lock: self._current_sensors = msg - def _on_button(self, msg: Bool) -> None: - """Callback for teleop button state.""" + def _on_button_bytes(self, data: bytes) -> None: + """Decode raw LCM bytes into Bool and update button state.""" + msg = Bool.lcm_decode(data) with self._lock: self._teleop_button = bool(msg.data) # ------------------------------------------------------------------------- - # Deno Bridge Server + # Embedded Web Server # ------------------------------------------------------------------------- def _start_server(self) -> None: - """Launch the Deno WebSocket-to-LCM bridge server as a subprocess.""" - if self._server_process is not None and self._server_process.poll() is None: - logger.warning("Deno bridge already running", pid=self._server_process.pid) - return - - if shutil.which("deno") is None: - logger.error( - "Deno is not installed. Install it with: curl -fsSL https://deno.land/install.sh | sh" - ) + """Start the embedded FastAPI server with HTTPS in a daemon thread.""" + if self._web_server_thread is not None and self._web_server_thread.is_alive(): + logger.warning("Web server already running") return - script = str(self._server_script) - cmd = [ - "deno", - "run", - "--allow-net", - "--allow-read", - "--allow-run", - "--allow-write", - "--unstable-net", - script, - ] - try: - self._server_process = subprocess.Popen(cmd) - logger.info(f"Deno bridge server started (pid {self._server_process.pid})") - except OSError as e: - logger.error(f"Failed to start Deno bridge: {e}") + self._web_server_thread = threading.Thread( + target=self._web_server.run, + kwargs={"ssl": True, "ssl_certs_dir": get_project_root() / "assets" / "teleop_certs"}, + daemon=True, + name="PhoneTeleopWebServer", + ) + self._web_server_thread.start() + logger.info(f"Phone teleop web server started on https://0.0.0.0:{self.config.server_port}") def _stop_server(self) -> None: - """Terminate the Deno bridge server subprocess.""" - if self._server_process is None or self._server_process.poll() is not None: - self._server_process = None - return - - logger.info("Stopping Deno bridge server", pid=self._server_process.pid) - self._server_process.send_signal(signal.SIGTERM) - try: - self._server_process.wait(timeout=3) - except subprocess.TimeoutExpired: - logger.warning( - "Deno bridge did not exit, sending SIGKILL", pid=self._server_process.pid - ) - self._server_process.kill() - try: - self._server_process.wait(timeout=5) - except subprocess.TimeoutExpired: - logger.error("Deno bridge did not exit after SIGKILL") - logger.info("Deno bridge server stopped") - self._server_process = None + """Shutdown the embedded web server.""" + self._web_server.shutdown() + if self._web_server_thread is not None: + self._web_server_thread.join(timeout=3) + self._web_server_thread = None + logger.info("Phone teleop web server stopped") # ------------------------------------------------------------------------- # Control Loop diff --git a/dimos/teleop/phone/web/static/index.html b/dimos/teleop/phone/web/static/index.html index 6fad23b6c8..5bad9c727e 100644 --- a/dimos/teleop/phone/web/static/index.html +++ b/dimos/teleop/phone/web/static/index.html @@ -165,7 +165,7 @@

Sensors