Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 135 additions & 100 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<div align="center">
<img width="1000" alt="banner_bordered_trimmed" src="https://github.com/user-attachments/assets/15283d94-ad95-42c9-abd5-6565a222a837" /> </a>
<h4 align="center">The Open-Source Framework for Robotic Intelligence</h4>
<h4 align="center">Program Atoms</h4>
<h4 align="center">The Agentive Operating System for Generalist Robotics</h4>


<br>
Expand All @@ -16,76 +17,115 @@
[![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square&logo=docker&logoColor=white)](https://www.docker.com/)

<p align="center">
<a href="#how-does-dimensional-work">Key Features</a> •
<a href="#how-do-i-get-started">How To Use</a> •
<a href="#contributing--building-from-source">Contributing</a> • <a href="#license">License</a>
<a href="#the-dimensional-framework">Features</a> •
<a href="#installation">Installation</a> •
<a href="#documentation">Documentation</a> •
<a href="#development">Development</a> •
<a href="#contributing">Contributing</a>
</p>

</div>

> \[!NOTE]
>
> **Active Beta: Expect Breaking Changes**
> ⚠️ **Alpha Pre-Release: Expect Breaking Changes** ⚠️

# What is Dimensional?
# The Dimensional Framework

DimOS is both a language-agnostic framework and a Python-first library for robot control. It has optional ROS integration and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors.
Dimensional is the open-source, universal operating system for generalist robotics. On DimOS, developers
can design, build, and run physical ("dimensional") applications that run on any humanoid, quadruped,
drone, or wheeled embodiment.

The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more.
**Programming physical robots is now as simple as programming digital software**: Composable, Modular, Repeatable.

# How do I get started?
Core Features:
- **Navigation:** Production navigation stack for any robot with lidar: SLAM, terrain analysis, collision
avoidance, route planning, exploration.
- **Dashboard:** The DimOS command center gives developers the tooling to debug, visualize, compose, and
test dimensional applications in real-time. Control your robot via waypoint, agent query, keyboard,
VR, more.
- **Modules:** Standalone components (equivalent to ROS nodes) that publish and subscribe to typed
In/Out streams that communicate over DimOS transports. The building blocks of Dimensional.
- **Agents (experimental):** DimOS agents understand physical space, subscribe to sensor streams, and call
**physical** tools. Emergence appears when agents have physical agency.
- **MCP (experimental):** Vibecode robots by giving your AI editor (Cursor, Claude Code) MCP access to run
physical commands (move forward 1 meter, jump, etc.).
- **Manipulation (unreleased)** Classical (OMPL, IK, GraspGen), Agentive (TAMP), and VLA-native manipulation stack runs out-of-the-box on any DimOS supported arm embodiment.
- **Transport/Middleware:** DimOS native Python transport supports LCM, DDS, and SHM, plus ROS 2.
- **Robot integrations:** We integrate with the majority of hardware OEMs and are moving fast to cover
them all. Supported and/or immediate roadmap:

### Installation
| Category | Platforms |
| --- | --- |
| Quadrupeds | Unitree Go2, Unitree B1, AGIBOT D1 Max/Pro, Dobot Rover |
| Drones | DJI Mavic 2, Holybro x500 |
| Humanoids | Unitree G1, Booster K1, AGIBOT X2, ABIBOT A2 |
| Arms | OpenARMs, xARM 6/7, AgileX Piper, HighTorque Pantera |

- Linux is supported, with tests being performed on Ubuntu 22.04 and 24.04
- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)*
- instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python`
# Getting Started

## Installation

Supported/tested matrix:

| Platform | Status | Tested | Required System deps |
| --- | --- | --- | --- |
| Linux | supported | Ubuntu 22.04, 24.04 | See below |
| macOS | experimental beta | not CI-tested | `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` |

Note: macOS is usable but expect inconsistent/flaky behavior (rather than hard errors/crashes).

```sh
sudo apt-get update
sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev
# install uv for python
curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
```

#
# NOTE!!! the first time, you're going to have an empty/black rerun window for a while
#
# the command needs to download the replay file (2.4gb), which takes a bit
Option 1: Install in a virtualenv

```sh

# OPTION 1: install dimos in a virtualenv
uv venv && . .venv/bin/activate
uv pip install 'dimos[base,unitree]'
# replay recorded data to test that the system is working
# IMPORTANT: First replay run will show a black rerun window while 2.4 GB downloads from LFS
dimos --replay run unitree-go2
```

Option 2: Run without installing

# OPTION 2: if you want to test out dimos without installing run:
```sh
uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2
```

<!-- command for testing pre launch: `GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" uv pip install 'dimos[unitree] @ git+ssh://git@github.com/dimensionalOS/dimos.git@dev'` -->

### Usage
### Test Installation

#### Control a robot in a simulation (no robot required)

After running the commads below, open http://localhost:7779/command-center to control the robot movement.

```sh
export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors
# ignore the warp warnings
dimos --viewer-backend rerun-web --simulation run unitree-go2
```

#### Get it working on a physical robot!
#### Control a real robot (Unitree Go2 over WebRTC)

```sh
export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE
export ROBOT_IP=<YOUR_ROBOT_IP>
dimos --viewer-backend rerun-web run unitree-go2
```

#### Have it controlled by AI!
After running dimOS open http://localhost:7779 to control robot movement.

WARNING: This is a demo showing the **connection** between AI and robotic control -- not a demo of a super-intelligent AI. Be ready to physically prevent your robot from taking dumb physical actions.
#### Dimensional Agents

> \[!NOTE]
>
> **Experimental Beta: Potential unstoppable robot sentience**

```sh
export OPENAI_API_KEY=<your private key>
Expand All @@ -97,126 +137,121 @@ After running that, open a new terminal and run the following to start giving in
# activate the venv in this new terminal
source .venv/bin/activate

# Note: after running the next command, WAIT for the agent to connect
# (this will take a while the first time)
# then tell the agent "explore the room"
# then tell it to go to something, ex: "go to the door"
humancli
```

# How do I use it as a library?
# The Dimensional Library

### Simple Camera Activation
### Modules

Assuming you have a webcam, save the following as a python file and run it:
Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages. See below a simple robot connection module that sends streams of continuous `cmd_vel` to the robot and recieves `color_image` to a simple `Listener` module.

```py
import threading, time, numpy as np
from dimos.core import In, Module, Out, rpc
from dimos.core.blueprints import autoconnect
from dimos.hardware.sensors.camera.module import CameraModule
from dimos.msgs.geometry_msgs import Twist
from dimos.msgs.sensor_msgs import Image
from dimos.msgs.sensor_msgs.image_impls.AbstractImage import ImageFormat

class RobotConnection(Module):
cmd_vel: In[Twist]
color_image: Out[Image]

@rpc
def start(self):
threading.Thread(target=self._image_loop, daemon=True).start()

def _image_loop(self):
while True:
img = Image.from_numpy(
np.zeros((120, 160, 3), np.uint8),
format=ImageFormat.RGB,
frame_id="camera_optical",
)
self.color_image.publish(img)
time.sleep(0.2)

class Listener(Module):
color_image: In[Image]

@rpc
def start(self):
self.color_image.subscribe(lambda img: print(f"image {img.width}x{img.height}"))

if __name__ == "__main__":
autoconnect(
# technically autoconnect is not needed because we only have 1 module
CameraModule.blueprint()
RobotConnection.blueprint(),
Listener.blueprint(),
).build().loop()
```

### Write A Custom Module
### Blueprints

Blueprints are how robots are constructed on Dimensional; instructions for how to construct and wire modules. You compose them with
`autoconnect(...)`, which connects streams by `(name, type)` and returns a `ModuleBlueprintSet`.

Lets convert the camera's image to grayscale.
Blueprints can be composed, remapped, and have transports overridden if `autoconnect()` fails due to conflicting variable names or `In[]` and `Out[]` message types.

A blueprint example that connects the image stream from a robot to an LLM Agent for reasoning and action execution.
```py
from dimos.core.blueprints import autoconnect
from dimos.core import In, Out, Module, rpc
from dimos.hardware.sensors.camera.module import CameraModule
from dimos.core.transport import LCMTransport
from dimos.msgs.sensor_msgs import Image
from dimos.robot.unitree.connection.go2 import go2_connection
from dimos.agents.agent import llm_agent

from reactivex.disposable import Disposable
blueprint = autoconnect(
go2_connection(),
llm_agent(),
).transports({("color_image", Image): LCMTransport("/color_image", Image)})

class Listener(Module):
# the CameraModule has an Out[Image] named "color_image"
# How do we know this? Just print(CameraModule.module_info().outputs)
# the name ("color_image") must match the CameraModule's output
color_image: In[Image] = None
grayscale_image: Out[Image] = None
# Run the blueprint
blueprint.build().loop()
```

def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
self.count = 0
# Development

@rpc
def start(self) -> None:
super().start()
def callback_func(img: Image) -> None:
self.count += 1
print(f"got frame {self.count}")
print(f"img.data.shape: {img.data.shape}")
self.grayscale_image.publish(img.to_grayscale())

unsubscribe_func = self.color_image.subscribe(callback_func)
# the unsubscribe_func be called when the module is stopped
self._disposables.add(Disposable(
unsubscribe_func
))
```sh
GIT_LFS_SKIP_SMUDGE=1 git clone -b dev https://github.com/dimensionalOS/dimos.git
cd dimos
```

@rpc
def stop(self) -> None:
super().stop()
Then pick one of two development paths:

if __name__ == "__main__":
autoconnect(
Listener.blueprint(),
CameraModule.blueprint(),
).build().loop()
Option A: Devcontainer
```sh
./bin/dev
```

Option B: Editable install with uv
```sh
uv venv && . .venv/bin/activate
uv pip install -e '.[base,dev]'
```

#### Note: Many More Examples in the [Examples Folder](./examples)

### How do custom modules work? (Example breakdown)

- Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module.
- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to).
- Note: if you are a power user thinking "so streams must be statically declared?" the answer is no, there are ways to perform dynamic connections, but for type-checking and human sanity the creation of dynamic stream connections are under an advanced API and should be used as a last resort.
- The `autoconnect` ties everything together:
- The CameraModule has an output of `color_image`
- The Listener has an input of `color_image`
- Autoconnect puts them together, and checks that their types are compatible (both are of type `Image`)
- How can we see what In/Out streams are provided by a module?
- Open a python repl (e.g. `python`)
- Import the module, ex: `from dimos.hardware.sensors.camera.module import CameraModule`
- Print the module outputs: `print(CameraModule.module_info().outputs)`
- Print the module inputs: `print(CameraModule.module_info().inputs)`
- Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())`
- What about `@rpc`?
- If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable.
- Rpc methods get called using threads, meaning two rpc methods can be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls.
- The start/stop methods always need to be an rpc because they are called externally.
For system deps, Nix setups, and testing, see `/docs/development/README.md`.

### Monitoring & Debugging

In addition to rerun logging, DimOS comes with a number of monitoring tools:
DimOS comes with a number of monitoring tools:
- Run `lcmspy` to see how fast messages are being published on streams.
- Run `skillspy` to see how skills are being called, how long they are running, which are active, etc.
- Run `agentspy` to see the agent's status over time.
- If you suspect there is a bug within DimOS itself, you can enable extreme logging by prefixing the dimos command with `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 `. Ex: `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos --replay run unitree-go2`


# How does Dimensional work?
# Documentation

Concepts:
- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes.
- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system.
- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other.
- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data).
- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI).
- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools.

## Contributing / Building From Source

For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments.
## Contributing

We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue.

# License

DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source.
14 changes: 7 additions & 7 deletions docs/package_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,27 +11,27 @@ uv init
Install:

```bash
uv add dimos[dev,cpu,sim]
uv add dimos[base,dev,unitree]
```

Test the Unitree Go2 robot in the simulator:

```bash
uv run dimos-robot --simulation run unitree-g1
uv run dimos --simulation run unitree-go2
```

Run your actual robot:

```bash
uv run dimos-robot --robot-ip=192.168.X.XXX run unitree-g1
uv run dimos --robot-ip=192.168.X.XXX run unitree-go2
```

### Without installing

With `uv` you can run tools without having to explicitly install:

```bash
uvx --from dimos dimos-robot --robot-ip=192.168.X.XXX run unitree-g1
uvx --from dimos[base,unitree] dimos --robot-ip=192.168.X.XXX run unitree-go2
```

## With `pip`
Expand All @@ -46,17 +46,17 @@ python -m venv .venv
Install:

```bash
pip install dimos[dev,cpu,sim]
pip install dimos[base,dev,unitree]
```

Test the Unitree Go2 robot in the simulator:

```bash
dimos-robot --simulation run unitree-g1
dimos --simulation run unitree-go2
```

Run your actual robot:

```bash
dimos-robot --robot-ip=192.168.X.XXX run unitree-g1
dimos --robot-ip=192.168.X.XXX run unitree-go2
```