diff --git a/README.md b/README.md index c2f9992abd..9ccc7de197 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@
banner_bordered_trimmed -

The Open-Source Framework for Robotic Intelligence

+

Program Atoms

+

The Agentive Operating System for Generalist Robotics


@@ -16,59 +17,94 @@ [![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square&logo=docker&logoColor=white)](https://www.docker.com/)

- Key Features • - How To Use • - Contributing • License + Features • + Installation • + Documentation • + Development • + Contributing

> \[!NOTE] > -> **Active Beta: Expect Breaking Changes** +> ⚠️ **Alpha Pre-Release: Expect Breaking Changes** ⚠️ -# What is Dimensional? +# The Dimensional Framework -DimOS is both a language-agnostic framework and a Python-first library for robot control. It has optional ROS integration and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. +Dimensional is the open-source, universal operating system for generalist robotics. On DimOS, developers +can design, build, and run physical ("dimensional") applications that run on any humanoid, quadruped, +drone, or wheeled embodiment. -The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. +**Programming physical robots is now as simple as programming digital software**: Composable, Modular, Repeatable. -# How do I get started? +Core Features: +- **Navigation:** Production navigation stack for any robot with lidar: SLAM, terrain analysis, collision + avoidance, route planning, exploration. +- **Dashboard:** The DimOS command center gives developers the tooling to debug, visualize, compose, and + test dimensional applications in real-time. Control your robot via waypoint, agent query, keyboard, + VR, more. +- **Modules:** Standalone components (equivalent to ROS nodes) that publish and subscribe to typed + In/Out streams that communicate over DimOS transports. The building blocks of Dimensional. +- **Agents (experimental):** DimOS agents understand physical space, subscribe to sensor streams, and call + **physical** tools. Emergence appears when agents have physical agency. +- **MCP (experimental):** Vibecode robots by giving your AI editor (Cursor, Claude Code) MCP access to run + physical commands (move forward 1 meter, jump, etc.). +- **Manipulation (unreleased)** Classical (OMPL, IK, GraspGen), Agentive (TAMP), and VLA-native manipulation stack runs out-of-the-box on any DimOS supported arm embodiment. +- **Transport/Middleware:** DimOS native Python transport supports LCM, DDS, and SHM, plus ROS 2. +- **Robot integrations:** We integrate with the majority of hardware OEMs and are moving fast to cover + them all. Supported and/or immediate roadmap: -### Installation + | Category | Platforms | + | --- | --- | + | Quadrupeds | Unitree Go2, Unitree B1, AGIBOT D1 Max/Pro, Dobot Rover | + | Drones | DJI Mavic 2, Holybro x500 | + | Humanoids | Unitree G1, Booster K1, AGIBOT X2, ABIBOT A2 | + | Arms | OpenARMs, xARM 6/7, AgileX Piper, HighTorque Pantera | -- Linux is supported, with tests being performed on Ubuntu 22.04 and 24.04 -- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* - - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` +# Getting Started + +## Installation + +Supported/tested matrix: + +| Platform | Status | Tested | Required System deps | +| --- | --- | --- | --- | +| Linux | supported | Ubuntu 22.04, 24.04 | See below | +| macOS | experimental beta | not CI-tested | `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` | + +Note: macOS is usable but expect inconsistent/flaky behavior (rather than hard errors/crashes). ```sh sudo apt-get update sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev # install uv for python curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" +``` -# -# NOTE!!! the first time, you're going to have an empty/black rerun window for a while -# -# the command needs to download the replay file (2.4gb), which takes a bit +Option 1: Install in a virtualenv + +```sh -# OPTION 1: install dimos in a virtualenv uv venv && . .venv/bin/activate uv pip install 'dimos[base,unitree]' # replay recorded data to test that the system is working +# IMPORTANT: First replay run will show a black rerun window while 2.4 GB downloads from LFS dimos --replay run unitree-go2 +``` + +Option 2: Run without installing -# OPTION 2: if you want to test out dimos without installing run: +```sh uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 ``` -### Usage +### Test Installation #### Control a robot in a simulation (no robot required) -After running the commads below, open http://localhost:7779/command-center to control the robot movement. ```sh export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors @@ -76,16 +112,20 @@ export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors dimos --viewer-backend rerun-web --simulation run unitree-go2 ``` -#### Get it working on a physical robot! +#### Control a real robot (Unitree Go2 over WebRTC) ```sh -export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +export ROBOT_IP= dimos --viewer-backend rerun-web run unitree-go2 ``` -#### Have it controlled by AI! +After running dimOS open http://localhost:7779 to control robot movement. -WARNING: This is a demo showing the **connection** between AI and robotic control -- not a demo of a super-intelligent AI. Be ready to physically prevent your robot from taking dumb physical actions. +#### Dimensional Agents + +> \[!NOTE] +> +> **Experimental Beta: Potential unstoppable robot sentience** ```sh export OPENAI_API_KEY= @@ -97,111 +137,113 @@ After running that, open a new terminal and run the following to start giving in # activate the venv in this new terminal source .venv/bin/activate -# Note: after running the next command, WAIT for the agent to connect -# (this will take a while the first time) # then tell the agent "explore the room" # then tell it to go to something, ex: "go to the door" humancli ``` -# How do I use it as a library? +# The Dimensional Library -### Simple Camera Activation +### Modules -Assuming you have a webcam, save the following as a python file and run it: +Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages. See below a simple robot connection module that sends streams of continuous `cmd_vel` to the robot and recieves `color_image` to a simple `Listener` module. ```py +import threading, time, numpy as np +from dimos.core import In, Module, Out, rpc from dimos.core.blueprints import autoconnect -from dimos.hardware.sensors.camera.module import CameraModule +from dimos.msgs.geometry_msgs import Twist +from dimos.msgs.sensor_msgs import Image +from dimos.msgs.sensor_msgs.image_impls.AbstractImage import ImageFormat + +class RobotConnection(Module): + cmd_vel: In[Twist] + color_image: Out[Image] + + @rpc + def start(self): + threading.Thread(target=self._image_loop, daemon=True).start() + + def _image_loop(self): + while True: + img = Image.from_numpy( + np.zeros((120, 160, 3), np.uint8), + format=ImageFormat.RGB, + frame_id="camera_optical", + ) + self.color_image.publish(img) + time.sleep(0.2) + +class Listener(Module): + color_image: In[Image] + + @rpc + def start(self): + self.color_image.subscribe(lambda img: print(f"image {img.width}x{img.height}")) if __name__ == "__main__": autoconnect( - # technically autoconnect is not needed because we only have 1 module - CameraModule.blueprint() + RobotConnection.blueprint(), + Listener.blueprint(), ).build().loop() ``` -### Write A Custom Module +### Blueprints + +Blueprints are how robots are constructed on Dimensional; instructions for how to construct and wire modules. You compose them with +`autoconnect(...)`, which connects streams by `(name, type)` and returns a `ModuleBlueprintSet`. -Lets convert the camera's image to grayscale. +Blueprints can be composed, remapped, and have transports overridden if `autoconnect()` fails due to conflicting variable names or `In[]` and `Out[]` message types. +A blueprint example that connects the image stream from a robot to an LLM Agent for reasoning and action execution. ```py from dimos.core.blueprints import autoconnect -from dimos.core import In, Out, Module, rpc -from dimos.hardware.sensors.camera.module import CameraModule +from dimos.core.transport import LCMTransport from dimos.msgs.sensor_msgs import Image +from dimos.robot.unitree.connection.go2 import go2_connection +from dimos.agents.agent import llm_agent -from reactivex.disposable import Disposable +blueprint = autoconnect( + go2_connection(), + llm_agent(), +).transports({("color_image", Image): LCMTransport("/color_image", Image)}) -class Listener(Module): - # the CameraModule has an Out[Image] named "color_image" - # How do we know this? Just print(CameraModule.module_info().outputs) - # the name ("color_image") must match the CameraModule's output - color_image: In[Image] = None - grayscale_image: Out[Image] = None +# Run the blueprint +blueprint.build().loop() +``` - def __init__(self, *args, **kwargs) -> None: - super().__init__(*args, **kwargs) - self.count = 0 +# Development - @rpc - def start(self) -> None: - super().start() - def callback_func(img: Image) -> None: - self.count += 1 - print(f"got frame {self.count}") - print(f"img.data.shape: {img.data.shape}") - self.grayscale_image.publish(img.to_grayscale()) - - unsubscribe_func = self.color_image.subscribe(callback_func) - # the unsubscribe_func be called when the module is stopped - self._disposables.add(Disposable( - unsubscribe_func - )) +```sh +GIT_LFS_SKIP_SMUDGE=1 git clone -b dev https://github.com/dimensionalOS/dimos.git +cd dimos +``` - @rpc - def stop(self) -> None: - super().stop() +Then pick one of two development paths: -if __name__ == "__main__": - autoconnect( - Listener.blueprint(), - CameraModule.blueprint(), - ).build().loop() +Option A: Devcontainer +```sh +./bin/dev +``` + +Option B: Editable install with uv +```sh +uv venv && . .venv/bin/activate +uv pip install -e '.[base,dev]' ``` -#### Note: Many More Examples in the [Examples Folder](./examples) - -### How do custom modules work? (Example breakdown) - -- Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. -- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). - - Note: if you are a power user thinking "so streams must be statically declared?" the answer is no, there are ways to perform dynamic connections, but for type-checking and human sanity the creation of dynamic stream connections are under an advanced API and should be used as a last resort. -- The `autoconnect` ties everything together: - - The CameraModule has an output of `color_image` - - The Listener has an input of `color_image` - - Autoconnect puts them together, and checks that their types are compatible (both are of type `Image`) -- How can we see what In/Out streams are provided by a module? - - Open a python repl (e.g. `python`) - - Import the module, ex: `from dimos.hardware.sensors.camera.module import CameraModule` - - Print the module outputs: `print(CameraModule.module_info().outputs)` - - Print the module inputs: `print(CameraModule.module_info().inputs)` - - Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())` -- What about `@rpc`? - - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. - - Rpc methods get called using threads, meaning two rpc methods can be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. - - The start/stop methods always need to be an rpc because they are called externally. +For system deps, Nix setups, and testing, see `/docs/development/README.md`. ### Monitoring & Debugging -In addition to rerun logging, DimOS comes with a number of monitoring tools: +DimOS comes with a number of monitoring tools: - Run `lcmspy` to see how fast messages are being published on streams. - Run `skillspy` to see how skills are being called, how long they are running, which are active, etc. - Run `agentspy` to see the agent's status over time. - If you suspect there is a bug within DimOS itself, you can enable extreme logging by prefixing the dimos command with `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 `. Ex: `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos --replay run unitree-go2` -# How does Dimensional work? +# Documentation Concepts: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. @@ -209,14 +251,7 @@ Concepts: - [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. - [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). - [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). -- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. - -## Contributing / Building From Source -For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. +## Contributing We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. - -# License - -DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. diff --git a/docs/package_usage.md b/docs/package_usage.md index 24584a2e79..328708122e 100644 --- a/docs/package_usage.md +++ b/docs/package_usage.md @@ -11,19 +11,19 @@ uv init Install: ```bash -uv add dimos[dev,cpu,sim] +uv add dimos[base,dev,unitree] ``` Test the Unitree Go2 robot in the simulator: ```bash -uv run dimos-robot --simulation run unitree-g1 +uv run dimos --simulation run unitree-go2 ``` Run your actual robot: ```bash -uv run dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 +uv run dimos --robot-ip=192.168.X.XXX run unitree-go2 ``` ### Without installing @@ -31,7 +31,7 @@ uv run dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 With `uv` you can run tools without having to explicitly install: ```bash -uvx --from dimos dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 +uvx --from dimos[base,unitree] dimos --robot-ip=192.168.X.XXX run unitree-go2 ``` ## With `pip` @@ -46,17 +46,17 @@ python -m venv .venv Install: ```bash -pip install dimos[dev,cpu,sim] +pip install dimos[base,dev,unitree] ``` Test the Unitree Go2 robot in the simulator: ```bash -dimos-robot --simulation run unitree-g1 +dimos --simulation run unitree-go2 ``` Run your actual robot: ```bash -dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 +dimos --robot-ip=192.168.X.XXX run unitree-go2 ```