Skip to content

Task/manipulation tests fixes#1522

Merged
spomichter merged 9 commits intodevfrom
task/manipulation-tests-fixes
Mar 11, 2026
Merged

Task/manipulation tests fixes#1522
spomichter merged 9 commits intodevfrom
task/manipulation-tests-fixes

Conversation

@mustafab0
Copy link
Contributor

Problem

Manipulation blueprints required running multiple dimos run commands in separate terminals (e.g. coordinator-mock + xarm7-planner-coordinator), which was confusing and incompatible with the daemon's single-blueprint limitation. The interactive RPC client was also missing, and pyrealsense2 wasn't listed as a manipulation dependency.


Solution

  • Renamed manipulation_blueprints.py to blueprints.py for consistency with other modules.
  • Baked the mock coordinator into xarm7_planner_coordinator via autoconnect, so it runs as a single dimos run xarm7-planner-coordinator command.
  • Added manipulation_client.py — an interactive Python REPL (python -i) for plan/preview/execute workflows against a running ManipulationModule.
  • Added pyrealsense2 to the manipulation dependency group in pyproject.toml.
  • Updated README quick start to reflect single-command blueprints.

Breaking Changes

  • dimos run xarm7-planner-coordinator no longer requires a separate dimos run coordinator-mock. If you were running the coordinator separately for this blueprint, you can stop doing that.

How to Test

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 11, 2026

Greptile Summary

This PR simplifies the manipulation workflow by renaming manipulation_blueprints.pyblueprints.py, baking the mock coordinator directly into xarm7_planner_coordinator via autoconnect (eliminating the need for a separate terminal), adding a new interactive manipulation_client.py RPC shell, and adding pyrealsense2 to the manipulation dependency group.

Key changes:

  • dimos/manipulation/blueprints.py: xarm7_planner_coordinator now self-contains a control_coordinator (mock adapter) via autoconnect — single-command launch is now functional
  • dimos/manipulation/planning/examples/manipulation_client.py: New interactive REPL for plan/preview/execute workflows; however, the home() function has a bug — it passes a raw list directly to plan_to_joints instead of wrapping it in a JointState (as plan() correctly does), and does not guard against get_robot_info() returning None
  • dimos/robot/all_blueprints.py + dimos/robot/manipulators/xarm/blueprints.py: Import paths updated to reflect the rename — correct and complete
  • pyproject.toml / uv.lock: pyrealsense2 added to manipulation extras — appropriate fix
  • unitree_sdk2_python: A git submodule was added that is entirely unrelated to this PR's stated goals and was not mentioned in the description — this should be reviewed for accidental inclusion

Confidence Score: 3/5

  • Safe to merge after fixing the home() bug in manipulation_client.py and confirming the unrelated unitree_sdk2_python submodule addition is intentional.
  • The core blueprint refactoring (rename + autoconnect for the mock coordinator) is clean and well-structured. The import path updates are mechanical and complete. However, the new manipulation_client.py contains a clear type mismatch bug in home() — it passes a raw list to plan_to_joints which expects a JointState, unlike the plan() helper which wraps correctly — and a null-safety issue on the same function. Additionally, an untracked unitree_sdk2_python git submodule appears in the diff with no mention in the PR description, which warrants explicit confirmation before merging.
  • dimos/manipulation/planning/examples/manipulation_client.py (home() bug) and unitree_sdk2_python (unexplained submodule addition)

Important Files Changed

Filename Overview
dimos/manipulation/blueprints.py Renamed from manipulation_blueprints.py; xarm7_planner_coordinator now self-contains the mock coordinator via autoconnect, eliminating the need for a separate dimos run coordinator-mock process. Logic is clean and consistent with how other blueprints in the codebase compose coordinators.
dimos/manipulation/planning/examples/manipulation_client.py New interactive RPC client. The home() function has two bugs: (1) it passes a raw list instead of a JointState to plan_to_joints — unlike plan() which correctly wraps the list — causing a runtime type mismatch; (2) it calls .get() on the result of get_robot_info() which can return None, leading to an AttributeError.
dimos/robot/all_blueprints.py Import paths updated from dimos.manipulation.manipulation_blueprints to dimos.manipulation.blueprints across all five affected blueprint keys. Change is mechanical and correct.
dimos/robot/manipulators/xarm/blueprints.py Import updated from manipulation_blueprints to blueprints. Purely a rename-tracking change with no functional impact.
pyproject.toml pyrealsense2 correctly added to the manipulation extras group, matching the RealSense camera dependency used in xarm_perception blueprints.
unitree_sdk2_python A unitree_sdk2_python git submodule was added at commit a035adea, but this is completely unrelated to the manipulation/blueprint changes described in the PR and was not mentioned in the PR description. Likely an accidental inclusion.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as dimos CLI
    participant XAP as xarm7_planner_coordinator<br/>(autoconnect)
    participant MM as ManipulationModule
    participant CC as control_coordinator<br/>(mock adapter)
    participant Client as manipulation_client.py<br/>(python -i)

    Note over CLI,XAP: Single command: dimos run xarm7-planner-coordinator
    CLI->>XAP: launch blueprint
    XAP->>MM: start ManipulationModule
    XAP->>CC: start control_coordinator (mock)
    MM-->>CC: LCMTransport /coordinator/joint_state

    Note over User,Client: Second terminal: python -i -m ...manipulation_client
    User->>Client: import / _client = RPCClient(None, ManipulationModule)
    Client->>MM: RPC connection (LCMRPC)

    User->>Client: plan([0.1]*7)
    Client->>MM: plan_to_joints(JointState)
    MM-->>Client: bool (success)

    User->>Client: preview()
    Client->>MM: preview_path(duration)
    MM-->>Client: Meshcat URL

    User->>Client: execute()
    Client->>MM: execute()
    MM->>CC: trajectory RPC (traj_arm)
    CC-->>MM: execution result
    MM-->>Client: bool (success)
Loading

Last reviewed commit: 00d626c

spomichter
spomichter previously approved these changes Mar 11, 2026
spomichter
spomichter previously approved these changes Mar 11, 2026
@spomichter spomichter merged commit bc9b4ff into dev Mar 11, 2026
11 checks passed
@spomichter spomichter deleted the task/manipulation-tests-fixes branch March 11, 2026 22:26
@spomichter spomichter mentioned this pull request Mar 11, 2026
1 task
spomichter added a commit that referenced this pull request Mar 12, 2026
Release v0.0.11

82 PRs, 10 contributors, 396 files changed.

This release brings a production CLI, MCP tooling, temporal memory, and first-class support for coding agents. Dask has been removed. The entire stack now runs from `dimos run` through `dimos stop`.

### Agent-Native Development

DimOS is now built to be driven by coding agents. Point OpenClaw, Claude Code, or Cursor at [AGENTS.md](AGENTS.md) and they can build, run, and debug Dimensional applications using the CLI and MCP interfaces directly.

- **AGENTS.md** — comprehensive onboarding doc: architecture, CLI reference, skill rules, blueprint quick-reference. Your agent reads this and starts coding.
- **MCP server** — all `@skill` methods exposed as HTTP tools. External agents call `dimos mcp call relative_move --arg forward=0.5` or connect via JSON-RPC.
- **MCP CLI** — `dimos mcp list-tools`, `dimos mcp call`, `dimos mcp status`, `dimos mcp modules`
- **Agent context logging** — MCP tool calls and agent messages logged to per-run JSONL for debugging and replay.

### CLI & Daemon

Full process lifecycle — no more Ctrl-C in tmux.

- `dimos run --daemon` — background execution with health checks and run registry
- `dimos stop [--force]` — graceful shutdown with SIGTERM → SIGKILL fallback
- `dimos restart` — replays the original CLI arguments
- `dimos status` — PID, blueprint, uptime, MCP port
- `dimos log -f` — structured per-run logs with follow, JSON output, filtering
- `dimos show-config` — resolved GlobalConfig with source tracing

### Temporal-Spatial Memory

Robots in physical space ingest hours of video and lidar. Temporal-spatial memory gives them a human-like understanding of the world — causal object relationships, entity tracking through time and physical space, and the ability to answer complex temporal queries:

*Who spends the most time in the kitchen? What time on average do I wake up? Which set of switches toggles the main lights? Who was at the office at 9am last Thursday?*

Traditional frame-level embeddings (CLIP, ViT) lose temporal context and don't scale beyond a handful of frames. Video transformers are expensive and don't operate in RGB-D. Dimensional agents work with video + lidar natively, tracking entities across hours and days.

```bash
dimos --replay --replay-dir unitree_go2_office_walk2 run unitree-go2-temporal-memory
```

### Interactive Viewer

Custom Rerun fork (`dimos-viewer`) is now the default. Click-to-navigate: click a point in the 3D view → PointStamped → A* planner → robot moves.

- Camera | 3D split layout on Go2, G1, and drone blueprints
- Native keyboard teleop in the viewer
- `--viewer rerun|rerun-web|rerun-connect|foxglove|none`

### Drone Support

Drone blueprints modernized to match Go2 composition pattern. `drone-basic` and `drone-agentic` work with replay, Rerun, and the full CLI.

```bash
dimos --replay run drone-basic
dimos --replay run drone-agentic
```

### More

- **Go2 fleet control** — multi-robot with `--robot-ips` (#1487)
- **Replay `--replay-dir`** — select dataset, loops by default (#1519, #1494)
- **Interactive install** — `curl -fsSL .../install.sh | bash` (#1395)
- **Nix on non-Debian Linux** (#1472)
- **Remove Dask** — native worker pool (#1365)
- **Remove asyncio dependency** (#1367)
- **Perceive loop** — continuous observation module for agents (#1411)
- **Worker resource monitor** — `dtop` TUI (#1378)
- **G1 agent wiring fix** (#1518)
- **Rerun rate limiting** — prevents viewer OOM on continuous streams (#1509, #1521)
- **RotatingFileHandler** — prevents unbounded log growth (#1492)
- **Test coverage** (#1397), draft PR CI skip (#1398), manipulation test fixes (#1522)

### Breaking Changes

- `--viewer-backend` renamed to `--viewer`
- Dask removed — blueprints using Dask workers need migration to native worker pool
- Default viewer changed from `rerun-web` to `rerun` (native dimos-viewer)

### Contributors

@spomichter, @PaulNechifor, @ruthwikdasyam, @summeryang, @MustafaBhadsorawala, @leshy, @sambull, @JeffHykin, @RadientBrain

## Contributor License Agreement

- [x] I have read and approved the [CLA](https://github.com/dimensionalOS/dimos/blob/main/CLA.md).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants