Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 31 additions & 16 deletions dimos/teleop/README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,33 @@
# Teleop Stack

Teleoperation modules for DimOS. Currently supports Meta Quest 3 VR controllers.
Teleoperation modules for DimOS. Supports Meta Quest 3 VR controllers and phone motion sensors.

## Architecture

```
Quest Browser (WebXR)
Quest/Phone Browser
PoseStamped + Joy via WebSocket
LCM-encoded binary via WebSocket
Deno Bridge (teleop_server.ts)
Embedded FastAPI Server (HTTPS)
LCM topics
Fingerprint-based message dispatch
QuestTeleopModule
│ WebXR → robot frame transform
│ Pose computation + button state packing
TeleopModule (Quest or Phone)
│ Frame transforms + pose/twist computation
PoseStamped / TwistStamped / Buttons outputs
```

Each teleop module embeds a `RobotWebInterface` (FastAPI + uvicorn) that:
- Serves the teleop web app at `/teleop`
- Accepts WebSocket connections at `/ws`
- Handles SSL certificate generation for HTTPS (required by mobile sensor APIs)

## Modules

### QuestTeleopModule
Base teleop module. Gets controller data, computes output poses, and publishes them. Default engage: hold primary button (X/A). Subclass to customize.
Base Quest teleop module. Gets controller data via WebSocket, computes output poses, and publishes them. Default engage: hold primary button (X/A). Subclass to customize.

### ArmTeleopModule
Toggle-based engage — press primary button once to engage, press again to disengage.
Expand All @@ -34,6 +38,12 @@ Outputs TwistStamped (linear + angular velocity) instead of PoseStamped.
### VisualizingTeleopModule
Adds Rerun visualization for debugging. Extends ArmTeleopModule (toggle engage).

### PhoneTeleopModule
Base phone teleop module. Receives orientation + gyro data from phone motion sensors, computes velocity commands from orientation deltas.

### SimplePhoneTeleop
Filters to mobile-base axes (linear.x, linear.y, angular.z) and publishes as `Twist`.

## Subclassing

`QuestTeleopModule` is designed for extension. Override these methods:
Expand All @@ -60,16 +70,14 @@ teleop/
│ ├── quest_teleop_module.py # Base Quest teleop module
│ ├── quest_extensions.py # ArmTeleop, TwistTeleop, VisualizingTeleop
│ ├── quest_types.py # QuestControllerState, Buttons
│ └── web/ # Deno bridge + WebXR client
│ ├── teleop_server.ts
│ └── static/index.html
│ └── web/
│ └── static/index.html # WebXR client
├── phone/
│ ├── phone_teleop_module.py # Base Phone teleop module
│ ├── phone_extensions.py # SimplePhoneTeleop
│ ├── blueprints.py # Pre-wired configurations
│ └── web/ # Deno bridge + mobile web app
│ ├── teleop_server.ts
│ └── static/index.html
│ └── web/
│ └── static/index.html # Mobile sensor web app
├── utils/
│ ├── teleop_transforms.py # WebXR → robot frame math
│ └── teleop_visualization.py # Rerun visualization helpers
Expand All @@ -78,4 +86,11 @@ teleop/

## Quick Start

See [Quest Web README](quest/web/README.md) for running the Deno bridge and connecting the Quest headset.
```bash
dimos run arm-teleop # Quest arm teleop
dimos run phone-go2-teleop # Phone → Go2
```

Open `https://<host-ip>:<port>/teleop` on device. Accept the self-signed certificate.
- Quest: port 8443
- Phone: port 8444
56 changes: 12 additions & 44 deletions dimos/teleop/phone/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,66 +5,34 @@ Teleoperation via smartphone motion sensors. Tilt to drive.
## Architecture

```
Phone Browser (DeviceOrientation + DeviceMotion)
|
| TwistStamped + Bool via WebSocket
v
Deno Bridge (teleop_server.ts)
|
| LCM topics
v
PhoneTeleopModule
| Orientation delta from home pose
| Gains -> velocity commands
v
TwistStamped / Twist outputs
Phone Browser ──WebSocket──→ Embedded HTTPS Server ──→ PhoneTeleopModule
(sensors + button) (port 8444) (delta → velocity)
```

## Modules
## Running

### PhoneTeleopModule
Base module. Receives raw sensor data and button state. On engage (button hold), captures home orientation and publishes deltas as TwistStamped. Launches the Deno bridge server automatically.
```bash
dimos run phone-go2-teleop # Go2
dimos run simple-phone-teleop # Generic ground robot
```

### SimplePhoneTeleop
Filters to mobile-base axes (linear.x, linear.y, angular.z) and publishes as `Twist` on `cmd_vel` for direct autoconnect wiring with any module that has `cmd_vel: In[Twist]`.
Open `https://<host-ip>:8444/teleop` on phone. Accept cert, allow sensors, connect, hold to drive.

## Subclassing

Override these methods:

| Method | Purpose |
|--------|---------|
| `_handle_engage()` | Customize engage/disengage logic |
| `_should_publish()` | Add conditions for when to publish |
| `_publish_msg()` | Change output format |

**Do not acquire `self._lock` in overrides.** The control loop already holds it.

## LCM Topics

| Topic | Type | Description |
|-------|------|-------------|
| `/phone_sensors` | TwistStamped | linear=(roll,pitch,yaw) deg, angular=(gyro) deg/s |
| `/phone_button` | Bool | Teleop engage button (1=held) |
| `/teleop/twist` | TwistStamped | Output velocity command |

## Running

```bash
dimos run phone-go2-teleop # Go2
dimos run simple-phone-teleop # Generic ground robot
```

Server starts on port `8444`. Open `https://<host-ip>:8444` on phone, accept the self-signed certificate, allow sensor permissions, connect, hold button to drive.
`self._lock` is already held — don't acquire it in overrides.

## File Structure

```
phone/
├── phone_teleop_module.py # Base phone teleop module
├── phone_teleop_module.py # Base module
├── phone_extensions.py # SimplePhoneTeleop
├── blueprints.py # Pre-wired configurations
└── web/
├── teleop_server.ts # Deno WSS-to-LCM bridge
└── static/index.html # Mobile web app
├── blueprints.py
└── web/static/index.html # Mobile web app
```
Loading
Loading