From b5b9f3078521d539ad9e1694d68b3cca24a29d6a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 12 Jan 2026 22:13:40 -0800 Subject: [PATCH 001/136] add readme --- README.md | 544 +++++++++++------------------------------------------- 1 file changed, 110 insertions(+), 434 deletions(-) diff --git a/README.md b/README.md index 9a74d63aa7..565c0d1689 100644 --- a/README.md +++ b/README.md @@ -20,483 +20,159 @@ ## What is Dimensional? -Dimensional is an open-source framework for building agentive generalist robots. DimOS allows off-the-shelf Agents to call tools/functions and read sensor/state data directly from ROS. +Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. -The framework enables neurosymbolic orchestration of Agents as generalized spatial reasoners/planners and Robot state/action primitives as functions. +# How do I get started? -The result: cross-embodied *"Dimensional Applications"* exceptional at generalization and robust at symbolic action execution. +## Installation -## DIMOS x Unitree Go2 (OUT OF DATE) +#### TLDR: +- install [uv for python](https://docs.astral.sh/uv/) (`curl -LsSf https://astral.sh/uv/install.sh | sh`) +- install a few system dependencies: `sudo apt-get install -y curl g++ portaudio19-dev git-lfs libjpegturbo` +- `uv pip install 'dimos[base]'` +- run `dimos run camera-demo` (assuming you have a webcam) -We are shipping a first look at the DIMOS x Unitree Go2 integration, allowing for off-the-shelf Agents() to "call" Unitree ROS2 Nodes and WebRTC action primitives, including: +#### Details / Requirements -- Navigation control primitives (move, reverse, spinLeft, spinRight, etc.) -- WebRTC control primitives (FrontPounce, FrontFlip, FrontJump, etc.) -- Camera feeds (image_raw, compressed_image, etc.) -- IMU data -- State information -- Lidar / PointCloud primitives -- Any other generic Unitree ROS2 topics - -### Features - -- **DimOS Agents** - - Agent() classes with planning, spatial reasoning, and Robot.Skill() function calling abilities. - - Integrate with any off-the-shelf hosted or local model: OpenAIAgent, ClaudeAgent, GeminiAgent 🚧, DeepSeekAgent 🚧, HuggingFaceRemoteAgent, HuggingFaceLocalAgent, etc. - - Modular agent architecture for easy extensibility and chaining of Agent output --> Subagents input. - - Agent spatial / language memory for location grounded reasoning and recall. - -- **DimOS Infrastructure** - - A reactive data streaming architecture using RxPY to manage real-time video (or other sensor input), outbound commands, and inbound robot state between the DimOS interface, Agents, and ROS2. - - Robot Command Queue to handle complex multi-step actions to Robot. - - Simulation bindings (Genesis, Isaacsim, etc.) to test your agentive application before deploying to a physical robot. - -- **DimOS Interface / Development Tools** - - Local development interface to control your robot, orchestrate agents, visualize camera/lidar streams, and debug your dimensional agentive application. - -## MacOS Installation +- Linux (MacOS support is in beta, you're welcome to roughly follow these steps, which will run on MacOS) ```sh -# Install Nix -curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install - -# clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git - -# setup the environment (follow the prompts after nix develop) -cd dimos -nix develop - -# You should be able to follow the instructions below as well for a more manual installation -``` - ---- -## Python Installation -Tested on Ubuntu 22.04/24.04 - -```bash -sudo apt install python3-venv - -# Clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git -cd dimos - -# Create and activate virtual environment -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# Install LFS -sudo apt install git-lfs -git lfs install - -# Install torch and torchvision if not already installed -# Example CUDA 11.7, Pytorch 2.0.1 (replace with your required pytorch version if different) -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -``` - -#### Install dependencies -```bash -# CPU only (reccomended to attempt first) -pip install -e '.[cpu,dev]' - -# CUDA install -pip install -e '.[cuda,dev]' - -# Copy and configure environment variables -cp default.env .env -``` - -#### Test the install -```bash -pytest -s dimos/ -``` - -#### Test Dimensional with a replay UnitreeGo2 stream (no robot required) -```bash +sudo apt-get update +sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh +# get access to the uv command +export PATH="$HOME/.local/bin:$PATH" +# create & activate a virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos +uv pip install 'dimos[base,unitree]' +# +# NOTE!!! you're going to have an empty/black rerun window for a while the first time +# +# this needs to download the replay file (2.4gb), which will take a while the first time dimos --replay run unitree-go2 ``` -#### Test Dimensional with a simulated UnitreeGo2 in MuJoCo (no robot required) -```bash -pip install -e '.[sim]' -export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors -dimos --simulation run unitree-go2 -``` - -#### Test Dimensional with a real UnitreeGo2 over WebRTC -```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2 -``` - -#### Test Dimensional with a real UnitreeGo2 running Agents -*OpenAI / Alibaba keys required* -```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2-agentic -``` ---- - -### Agent API keys - -Full functionality will require API keys for the following: + -Requirements: -- OpenAI API key (required for all LLMAgents due to OpenAIEmbeddings) -- Claude API key (required for ClaudeAgent) -- Alibaba API key (required for Navigation skills) +#### Get it working on a real physical robot! -These keys can be added to your .env file or exported as environment variables. -``` -export OPENAI_API_KEY= -export CLAUDE_API_KEY= -export ALIBABA_API_KEY= +```sh +export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +dimos run unitree-go2 ``` -### ROS2 Unitree Go2 SDK Installation + -#### System Requirements -- Ubuntu 22.04 -- ROS2 Distros: Iron, Humble, Rolling - -See [Unitree Go2 ROS2 SDK](https://github.com/dimensionalOS/go2_ros2_sdk) for additional installation instructions. +#### Get it working in an interactive simulation! ```bash -mkdir -p ros2_ws -cd ros2_ws -git clone --recurse-submodules https://github.com/dimensionalOS/go2_ros2_sdk.git src -sudo apt install ros-$ROS_DISTRO-image-tools -sudo apt install ros-$ROS_DISTRO-vision-msgs - -sudo apt install python3-pip clang portaudio19-dev -cd src -pip install -r requirements.txt -cd .. - -# Ensure clean python install before running -source /opt/ros/$ROS_DISTRO/setup.bash -rosdep install --from-paths src --ignore-src -r -y -colcon build +uv pip install 'dimos[sim]' +uv pip install pygame +export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors +# ignore the warp warnings +dimos --simulation run unitree-go2 --extra-module keyboard_teleop +# open http://localhost:7779/command-center in your browser ``` + + -### Run the test application -#### ROS2 Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -source /opt/ros/$ROS_DISTRO/setup.bash - -export ROBOT_IP="robot_ip" #for muliple robots, just split by , -export CONN_TYPE="webrtc" -ros2 launch go2_robot_sdk robot.launch.py - -``` - -#### Python Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -python tests/run.py -``` +#### Have it controlled by AI! -#### DimOS Interface: ```bash -cd dimos/web/dimos_interface -yarn install -yarn dev # you may need to run sudo if previously built via Docker -``` - -### Project Structure (OUT OF DATE) - -``` -. -├── dimos/ -│ ├── agents/ # Agent implementations -│ │ └── memory/ # Memory systems for agents, including semantic memory -│ ├── environment/ # Environment context and sensing -│ ├── hardware/ # Hardware abstraction and interfaces -│ ├── models/ # ML model definitions and implementations -│ │ ├── Detic/ # Detic object detection model -│ │ ├── depth/ # Depth estimation models -│ │ ├── segmentation/ # Image segmentation models -│ ├── perception/ # Computer vision and sensing -│ │ ├── detection2d/ # 2D object detection -│ │ └── segmentation/ # Image segmentation pipelines -│ ├── robot/ # Robot control and hardware interface -│ │ ├── global_planner/ # Path planning at global scale -│ │ ├── local_planner/ # Local navigation planning -│ │ └── unitree/ # Unitree Go2 specific implementations -│ ├── simulation/ # Robot simulation environments -│ │ ├── genesis/ # Genesis simulation integration -│ │ └── isaac/ # NVIDIA Isaac Sim integration -│ ├── skills/ # Task-specific robot capabilities -│ │ └── rest/ # REST API based skills -│ ├── stream/ # WebRTC and data streaming -│ │ ├── audio/ # Audio streaming components -│ │ └── video_providers/ # Video streaming components -│ ├── types/ # Type definitions and interfaces -│ ├── utils/ # Utility functions and helpers -│ └── web/ # DimOS development interface and API -│ ├── dimos_interface/ # DimOS web interface -│ └── websocket_vis/ # Websocket visualizations -├── tests/ # Test files -│ ├── genesissim/ # Genesis simulator tests -│ └── isaacsim/ # Isaac Sim tests -└── docker/ # Docker configuration files - ├── agent/ # Agent service containers - ├── interface/ # Interface containers - ├── simulation/ # Simulation environment containers - └── unitree/ # Unitree robot specific containers -``` - -## Building - -### Simple DimOS Application (OUT OF DATE) - -```python -from dimos.robot.unitree.unitree_go2 import UnitreeGo2 -from dimos.robot.unitree.unitree_skills import MyUnitreeSkills -from dimos.robot.unitree.unitree_ros_control import UnitreeROSControl -from dimos.agents_deprecated.agent import OpenAIAgent - -# Initialize robot -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) - -# Initialize agent -agent = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_video_stream=robot.get_ros_video_stream(), - skills=robot.get_skills(), - system_query="Jump when you see a human! Front flip when you see a dog!", - model_name="gpt-4o" - ) - -while True: # keep process running - time.sleep(1) -``` - - -### DimOS Application with Agent chaining (OUT OF DATE) - -Let's build a simple DimOS application with Agent chaining. We define a ```planner``` as a ```PlanningAgent``` that takes in user input to devise a complex multi-step plan. This plan is passed step-by-step to an ```executor``` agent that can queue ```AbstractRobotSkill``` commands to the ```ROSCommandQueue```. - -Our reactive Pub/Sub data streaming architecture allows for chaining of ```Agent_0 --> Agent_1 --> ... --> Agent_n``` via the ```input_query_stream``` parameter in each which takes an ```Observable``` input from the previous Agent in the chain. - -**Via this method you can chain together any number of Agents() to create complex dimensional applications.** - -```python - -web_interface = RobotWebInterface(port=5555) - -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) - -# Initialize master planning agent -planner = PlanningAgent( - dev_name="UnitreePlanningAgent", - input_query_stream=web_interface.query_stream, # Takes user input from dimOS interface - skills=robot.get_skills(), - model_name="gpt-4o", - ) - -# Initialize execution agent -executor = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_query_stream=planner.get_response_observable(), # Takes planner output as input - skills=robot.get_skills(), - model_name="gpt-4o", - system_query=""" - You are a robot execution agent that can execute tasks on a virtual - robot. ONLY OUTPUT THE SKILLS TO EXECUTE. - """ - ) - -while True: # keep process running - time.sleep(1) -``` - -### Calling Action Primitives (OUT OF DATE) - -Call action primitives directly from ```Robot()``` for prototyping and testing. - -```python -robot = UnitreeGo2(ip=robot_ip,) - -# Call a Unitree WebRTC action primitive -robot.webrtc_req(api_id=1016) # "Hello" command - -# Call a ROS2 action primitive -robot.move(distance=1.0, speed=0.5) -``` - -### Creating Custom Skills (non-unitree specific) - -#### Create basic custom skills by inheriting from ```AbstractRobotSkill``` and implementing the ```__call__``` method. - -```python -class Move(AbstractRobotSkill): - distance: float = Field(...,description="Distance to reverse in meters") - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - return self._robot.move(distance=self.distance) -``` - -#### Chain together skills to create recursive skill trees - -```python -class JumpAndFlip(AbstractRobotSkill): - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - jump = Jump(robot=self._robot) - flip = Flip(robot=self._robot) - return (jump() and flip()) +export OPENAI_API_KEY= +dimos run unitree-go2-agentic +# open the following in a different terminal tab +humancli ``` + + -### Integrating Skills with Agents: Single Skills and Skill Libraries - -DimOS agents, such as `OpenAIAgent`, can be endowed with capabilities through two primary mechanisms: by providing them with individual skill classes or with comprehensive `SkillLibrary` instances. This design offers flexibility in how robot functionalities are defined and managed within your agent-based applications. - -**Agent's `skills` Parameter** - -The `skills` parameter in an agent's constructor is key to this integration: - -1. **A Single Skill Class**: This approach is suitable for skills that are relatively self-contained or have straightforward initialization requirements. - * You pass the skill *class itself* (e.g., `GreeterSkill`) directly to the agent's `skills` parameter. - * The agent then takes on the responsibility of instantiating this skill when it's invoked. This typically involves the agent providing necessary context to the skill's constructor (`__init__`), such as a `Robot` instance (or any other private instance variable) if the skill requires it. + -2. **A `SkillLibrary` Instance**: This is the preferred method for managing a collection of skills, especially when skills have dependencies, require specific configurations, or need to share parameters. - * You first define your custom skill library by inheriting from `SkillLibrary`. Then, you create and configure an *instance* of this library (e.g., `my_lib = EntertainmentSkills(robot=robot_instance)`). - * This pre-configured `SkillLibrary` instance is then passed to the agent's `skills` parameter. The library itself manages the lifecycle and provision of its contained skills. + + + -**Examples:** -#### 1. Using a Single Skill Class with an Agent +# How do I use it as a library? -First, define your skill. For instance, a `GreeterSkill` that can deliver a configurable greeting: -```python -class GreeterSkill(AbstractSkill): - """Greats the user with a friendly message.""" # Gives the agent better context for understanding (the more detailed the better). +Simple camera activation (save this as a python file and run it): - greeting: str = Field(..., description="The greating message to display.") # The field needed for the calling of the function. Your agent will also pull from the description here to gain better context. +```py +from dimos.core.blueprints import autoconnect +from dimos.hardware.camera.module import CameraModule - def __init__(self, greeting_message: Optional[str] = None, **data): - super().__init__(**data) - if greeting_message: - self.greeting = greeting_message - # Any additional skill-specific initialization can go here - - def __call__(self): - super().__call__() # Call parent's method if it contains base logic - # Implement the logic for the skill - print(self.greeting) - return f"Greeting delivered: '{self.greeting}'" -``` - -Next, register this skill *class* directly with your agent. The agent can then instantiate it, potentially with specific configurations if your agent or skill supports it (e.g., via default parameters or a more advanced setup). - -```python -agent = OpenAIAgent( - dev_name="GreetingBot", - system_query="You are a polite bot. If a user asks for a greeting, use your GreeterSkill.", - skills=GreeterSkill, # Pass the GreeterSkill CLASS - # The agent will instantiate GreeterSkill. - # If the skill had required __init__ args not provided by the agent automatically, - # this direct class passing might be insufficient without further agent logic - # or by passing a pre-configured instance (see SkillLibrary example). - # For simple skills like GreeterSkill with defaults or optional args, this works well. - model_name="gpt-4o" -) +if __name__ == "__main__": + autoconnect( + CameraModule.blueprint() + ).build().loop() + print("Webcam pipeline running. Press Ctrl+C to stop.") ``` -In this setup, when the `GreetingBot` agent decides to use the `GreeterSkill`, it will instantiate it. If the `GreeterSkill` were to be instantiated by the agent with a specific `greeting_message`, the agent's design would need to support passing such parameters during skill instantiation. -#### 2. Using a `SkillLibrary` Instance with an Agent +Write your own custom module: -Define the SkillLibrary and any skills it will manage in its collection: -```python -class MovementSkillsLibrary(SkillLibrary): - """A specialized skill library containing movement and navigation related skills.""" +```py +from dimos.core.blueprints import autoconnect +from dimos.core import In, Module, pSHMTransport +from dimos.core.core import rpc +from dimos.hardware.camera.module import CameraModule +from dimos.msgs.sensor_msgs import Image - def __init__(self, robot=None): - super().__init__() - self._robot = robot +from reactivex.disposable import Disposable - def initialize_skills(self, robot=None): - """Initialize all movement skills with the robot instance.""" - if robot: - self._robot = robot +class Listener(Module): + # the CameraModule has an Out[Image] named "color_image" + # this module will only receive those messages if + # the names ("color_image") match, otherwise autoconnect + # will not be able to connect one module to another + color_image: In[Image] = None + grayscale_image: Out[Image] = None - if not self._robot: - raise ValueError("Robot instance is required to initialize skills") + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.count = 0 - # Initialize with all movement-related skills - self.add(Navigate(robot=self._robot)) - self.add(NavigateToGoal(robot=self._robot)) - self.add(FollowHuman(robot=self._robot)) - self.add(NavigateToObject(robot=self._robot)) - self.add(GetPose(robot=self._robot)) # Position tracking skill -``` + @rpc + def start(self) -> None: + super().start() + def callback_func(img: Image) -> None: + self.count += 1 + print(f"got frame {self.count}") + print(f"img.data.shape: {img.data.shape}") + self.grayscale_image.publish(img.to_grayscale()) -Note the addision of initialized skills added to this collection above. + unsubscribe_func = self.color_image.subscribe(callback_func) + # disposables will be called when the module is stopped + self._disposables.add(Disposable( + unsubscribe_func + )) -Proceed to use this skill library in an Agent: + @rpc + def stop(self) -> None: + super().stop() -Finally, in your main application code: -```python -# 1. Create an instance of your custom skill library, configured with the robot -my_movement_skills = MovementSkillsLibrary(robot=robot_instance) - -# 2. Pass this library INSTANCE to the agent -performing_agent = OpenAIAgent( - dev_name="ShowBot", - system_query="You are a show robot. Use your skills as directed.", - skills=my_movement_skills, # Pass the configured SkillLibrary INSTANCE - model_name="gpt-4o" -) +if __name__ == "__main__": + autoconnect( + Listener.blueprint(), + CameraModule.blueprint(), + ).build().loop() ``` -### Unitree Test Files -- **`tests/run_go2_ros.py`**: Tests `UnitreeROSControl(ROSControl)` initialization in `UnitreeGo2(Robot)` via direct function calls `robot.move()` and `robot.webrtc_req()` -- **`tests/simple_agent_test.py`**: Tests a simple zero-shot class `OpenAIAgent` example -- **`tests/unitree/test_webrtc_queue.py`**: Tests `ROSCommandQueue` via a 20 back-to-back WebRTC requests to the robot -- **`tests/test_planning_agent_web_interface.py`**: Tests a simple two-stage `PlanningAgent` chained to an `ExecutionAgent` with backend FastAPI interface. -- **`tests/test_unitree_agent_queries_fastapi.py`**: Tests a zero-shot `ExecutionAgent` with backend FastAPI interface. - -## Documentation - -For detailed documentation, please visit our [documentation site](#) (Coming Soon). - -## Contributing - -We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. - -## License - -This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. - -## Acknowledgments - -Huge thanks to! -- The Roboverse Community and their unitree-specific help. Check out their [Discord](https://discord.gg/HEXNMCNhEh). -- @abizovnuralem for his work on the [Unitree Go2 ROS2 SDK](https://github.com/abizovnuralem/go2_ros2_sdk) we integrate with for DimOS. -- @legion1581 for his work on the [Unitree Go2 WebRTC Connect](https://github.com/legion1581/go2_webrtc_connect) from which we've pulled the ```Go2WebRTCConnection``` class and other types for seamless WebRTC-only integration with DimOS. -- @tfoldi for the webrtc_req integration via Unitree Go2 ROS2 SDK, which allows for seamless usage of Unitree WebRTC control primitives with DimOS. +### Note: Many More Examples in the [Examples Folder](https://github.com/dimensionalOS/dimos/examples) -## Contact -- GitHub Issues: For bug reports and feature requests -- Email: [build@dimensionalOS.com](mailto:build@dimensionalOS.com) +# How does DimOS work conceptually? -## Known Issues -- Agent() failure to execute Nav2 action primitives (move, reverse, spinLeft, spinRight) is almost always due to the internal ROS2 collision avoidance, which will sometimes incorrectly display obstacles or be overly sensitive. Look for ```[behavior_server]: Collision Ahead - Exiting DriveOnHeading``` in the ROS logs. Reccomend restarting ROS2 or moving robot from objects to resolve. -- ```docker-compose up --build``` does not fully initialize the ROS2 environment due to ```std::bad_alloc``` errors. This will occur during continuous docker development if the ```docker-compose down``` is not run consistently before rebuilding and/or you are on a machine with less RAM, as ROS is very memory intensive. Reccomend running to clear your docker cache/images/containers with ```docker system prune``` and rebuild. +There are several tools: +- Modules: parallel processing, several modules run. Modules are usually python classes but its possible to use C++, Rust, Typescript, etc to write a module. +- Streams: How modules communicate, a Pub / Sub system. +- Blueprints: a way to group modules together and define their connections to each other +- RPC: how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- Skills: Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools From 0f6dc00337e44da08595bb29175c15e98c8a2ad7 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 12 Jan 2026 22:15:08 -0800 Subject: [PATCH 002/136] remove unneeded pip install commands --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 565c0d1689..cd9472ddf8 100644 --- a/README.md +++ b/README.md @@ -68,8 +68,6 @@ dimos run unitree-go2 #### Get it working in an interactive simulation! ```bash -uv pip install 'dimos[sim]' -uv pip install pygame export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors # ignore the warp warnings dimos --simulation run unitree-go2 --extra-module keyboard_teleop From 742119355b16305596c63ad6f2cbf43409c5640e Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:11:43 -0800 Subject: [PATCH 003/136] add feedback --- README.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/README.md b/README.md index cd9472ddf8..46fbeb02ca 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,6 @@ Dimensional is an open-source framework for adding customized general intelligen ## Installation -#### TLDR: -- install [uv for python](https://docs.astral.sh/uv/) (`curl -LsSf https://astral.sh/uv/install.sh | sh`) -- install a few system dependencies: `sudo apt-get install -y curl g++ portaudio19-dev git-lfs libjpegturbo` -- `uv pip install 'dimos[base]'` -- run `dimos run camera-demo` (assuming you have a webcam) - #### Details / Requirements - Linux (MacOS support is in beta, you're welcome to roughly follow these steps, which will run on MacOS) @@ -40,9 +34,7 @@ Dimensional is an open-source framework for adding customized general intelligen sudo apt-get update sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev # install uv for python -curl -LsSf https://astral.sh/uv/install.sh | sh -# get access to the uv command -export PATH="$HOME/.local/bin:$PATH" +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" # create & activate a virtualenv (needed for dimos) uv venv && . .venv/bin/activate # install dimos From 2133e57dc7d58dad5ebf428ed026aed684df22a6 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:12:10 -0800 Subject: [PATCH 004/136] - --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 46fbeb02ca..b87c44f3e5 100644 --- a/README.md +++ b/README.md @@ -62,8 +62,8 @@ dimos run unitree-go2 ```bash export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors # ignore the warp warnings -dimos --simulation run unitree-go2 --extra-module keyboard_teleop -# open http://localhost:7779/command-center in your browser +dimos --simulation run unitree-go2 +# open http://localhost:7779/command-center in your browser to control the robot movement ``` From 932664e9e88067b9c4226e32d108c534475c5992 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:22:35 -0800 Subject: [PATCH 005/136] - --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b87c44f3e5..64e50e6c53 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -![Screenshot 2025-02-18 at 16-31-22 DimOS Terminal](/assets/dimos_terminal.png) + + # The Dimensional Framework *The universal framework for AI-native generalist robotics* ## What is Dimensional? +#### NOTE: This is a pre-release + Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. # How do I get started? From 8bc02f71d6999004851e71c22ba9186cff153efa Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:23:27 -0800 Subject: [PATCH 006/136] add warning --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 64e50e6c53..f12a1fe68e 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ ## What is Dimensional? -#### NOTE: This is a pre-release +#### Warning: This is a pre-release version Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From 923b0d1d4c9380eb2e43fa0bf95b77c3dc966fd4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:23:59 -0800 Subject: [PATCH 007/136] fix exmaples link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f12a1fe68e..f38b5bfe21 100644 --- a/README.md +++ b/README.md @@ -157,7 +157,7 @@ if __name__ == "__main__": ).build().loop() ``` -### Note: Many More Examples in the [Examples Folder](https://github.com/dimensionalOS/dimos/examples) +### Note: Many More Examples in the [Examples Folder](https://github.com/dimensionalOS/dimos/tree/main/examples) # How does DimOS work conceptually? From 831e2320a546da3c165f0745216a4bfe773d6a6b Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:26:10 -0800 Subject: [PATCH 008/136] enhance message about humancli, add crash warning --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f38b5bfe21..8d70fd13c2 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,10 @@ dimos --simulation run unitree-go2 ```bash export OPENAI_API_KEY= dimos run unitree-go2-agentic -# open the following in a different terminal tab +# open the following in a different terminal tab to tell it where to go +# Warning!: make sure to watch the bot, this is a pre-release it will run into stuff +# and get tangled/trapped +# ex: tell it to explore the room, then tell it to go to where it can see a door humancli ``` From 54990a2de3984efc2f10285befac7e77b80186d1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 15:28:28 -0800 Subject: [PATCH 009/136] - --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8d70fd13c2..251516b3e6 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,8 @@ Dimensional is an open-source framework for adding customized general intelligen #### Details / Requirements -- Linux (MacOS support is in beta, you're welcome to roughly follow these steps, which will run on MacOS) +- Linux, tested on Ubuntu 22.04, 24.04 +- MacOS support is in beta, you're welcome to roughly follow these steps, which will run on MacOS ```sh sudo apt-get update From 30a399fb477d78f0860f632e1b255d41312fdf99 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 22:52:30 -0800 Subject: [PATCH 010/136] add dev/contributing docs --- README.md | 64 ++++++++++++++++++- README_installation.md | 136 ----------------------------------------- 2 files changed, 63 insertions(+), 137 deletions(-) delete mode 100644 README_installation.md diff --git a/README.md b/README.md index 251516b3e6..08338b2c17 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,8 @@ Dimensional is an open-source framework for adding customized general intelligen #### Details / Requirements - Linux, tested on Ubuntu 22.04, 24.04 -- MacOS support is in beta, you're welcome to roughly follow these steps, which will run on MacOS +- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* + - instead of the apt-get command below run: `xcode-select --install && brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` ```sh sudo apt-get update @@ -173,3 +174,64 @@ There are several tools: - RPC: how one module can call a method on another module (arguments get serialized to JSON-like binary data) - Skills: Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools + +# How do I Contribute / Hack on DimOS? + +## Typical Installation + +Very similar to a normal installation, here are the steps to get the repo and get it running for development. + +```bash +git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +cd dimos +sudo apt-get update +sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" +# create & activate a virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos +uv pip install -e '.[base,dev,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + + + +Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: + +- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector +- **contact_graspnet_pytorch**: Required for robotic grasp prediction + +You can install them with: + +```bash +uv add git+https://github.com/openai/CLIP.git +uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git +uv add git+https://github.com/facebookresearch/detectron2.git +``` diff --git a/README_installation.md b/README_installation.md deleted file mode 100644 index ccdcb1a550..0000000000 --- a/README_installation.md +++ /dev/null @@ -1,136 +0,0 @@ -# DimOS - -## Installation - -Clone the repo: - -```bash -git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git -cd dimos -``` - -### System dependencies - -Tested on Ubuntu 22.04/24.04. - -```bash -sudo apt update -sudo apt install git-lfs python3-venv python3-pyaudio portaudio19-dev libturbojpeg0-dev -``` - -### Python dependencies - -Install `uv` by [following their instructions](https://docs.astral.sh/uv/getting-started/installation/) or just run: - -```bash -curl -LsSf https://astral.sh/uv/install.sh | sh -``` - -Install Python dependencies: - -```bash -uv sync -``` - -Depending on what you want to test you might want to install more optional dependencies as well (recommended): - -```bash -uv sync --extra dev --extra cpu --extra sim --extra drone -``` - -### Install Foxglove Studio (robot visualization and control) - -> **Note:** This will be obsolete once we finish our migration to open source [Rerun](https://rerun.io/). - -Download and install [Foxglove Studio](https://foxglove.dev/download): - -```bash -wget https://get.foxglove.dev/desktop/latest/foxglove-studio-latest-linux-amd64.deb -sudo apt install ./foxglove-studio-*.deb -``` - -[Register an account](https://app.foxglove.dev/signup) to use it. - -Open Foxglove Studio: - -```bash -foxglove-studio -``` - -To connect and load our dashboard: - -1. Click on "Open connection" -2. In the popup window, leave the WebSocket URL as `ws://localhost:8765` and click "Open" -3. In the top right, click on the "Default" dropdown, then "Import from file..." -4. Navigate to the `dimos` repo and select `assets/foxglove_dashboards/unitree.json` - -### Test the install - -Run the Python tests: - -```bash -uv run pytest dimos -``` - -They should all pass in about 3 minutes. - -### Test a robot replay - -Run the system by playing back recorded data from a robot (the replay data is automatically downloaded via Git LFS): - -```bash -uv run dimos --replay run unitree-go2-basic -``` - -You can visualize the robot data in Foxglove Studio. - -### Run a simulation - -```bash -uv run dimos --simulation run unitree-go2-basic -``` - -This will open a MuJoCo simulation window. You can also visualize data in Foxglove. - -If you want to also teleoperate the simulated robot run: - -```bash -uv run dimos --simulation run unitree-go2-basic --extra-module keyboard_teleop -``` - -This will also open a Keyboard Teleop window. Focus on the window and use WASD to control the robot. - -### Command center - -You can also control the robot from the `command-center` extension to Foxglove. - -First, pull the LFS file: - -```bash -git lfs pull --include="assets/dimensional.command-center-extension-0.0.1.foxe" -``` - -To install it, drag that file over the Foxglove Studio window. The extension will be installed automatically. Then, click on the "Add panel" icon on the top right and add "command-center". - -You can now click on the map to give it a travel goal, or click on "Start Keyboard Control" to teleoperate it. - -### Using `dimos` in your code - -If you want to use dimos in your own project (not the cloned repo), you can install it as a dependency: - -```bash -uv add dimos -``` - -Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: - -- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector -- **contact_graspnet_pytorch**: Required for robotic grasp prediction - -You can install them with: - -```bash -uv add git+https://github.com/openai/CLIP.git -uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git -uv add git+https://github.com/facebookresearch/detectron2.git -``` From b9b454694971024dd0cecfc7d482cfeb1bb50ec4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Tue, 13 Jan 2026 22:53:24 -0800 Subject: [PATCH 011/136] add license --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 08338b2c17..9a1ef0cf93 100644 --- a/README.md +++ b/README.md @@ -235,3 +235,7 @@ uv add git+https://github.com/openai/CLIP.git uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git uv add git+https://github.com/facebookresearch/detectron2.git ``` + +# License + +DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. From d68884570fcc112c6098146683de0555b913888c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 16:12:34 -0800 Subject: [PATCH 012/136] add uvx run --- README.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 9a1ef0cf93..cc48e95135 100644 --- a/README.md +++ b/README.md @@ -40,18 +40,22 @@ sudo apt-get update sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev # install uv for python curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" -# create & activate a virtualenv (needed for dimos) -uv venv && . .venv/bin/activate -# install dimos -uv pip install 'dimos[base,unitree]' + # -# NOTE!!! you're going to have an empty/black rerun window for a while the first time +# NOTE!!! the first time, you're going to have an empty/black rerun window for a while # -# this needs to download the replay file (2.4gb), which will take a while the first time +# the command needs to download the replay file (2.4gb), which takes a bit + +# OPTION 1: use without installing dimos +uvx --from 'dimos[base,unitree]' dimos --replay run unitree-g2 + +# OPTION 2: install dimos in a virtualenv +uv venv && . .venv/bin/activate +uv pip install 'dimos[base,unitree]' dimos --replay run unitree-go2 ``` - + #### Get it working on a real physical robot! From 55aa17417031667e5e2cdcb8711421d36dd2480d Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 16:21:03 -0800 Subject: [PATCH 013/136] consolidate package usage into readme.md --- docs/package_usage.md | 62 ------------------------------------------- 1 file changed, 62 deletions(-) delete mode 100644 docs/package_usage.md diff --git a/docs/package_usage.md b/docs/package_usage.md deleted file mode 100644 index 24584a2e79..0000000000 --- a/docs/package_usage.md +++ /dev/null @@ -1,62 +0,0 @@ -# Package Usage - -## With `uv` - -Init your repo if not already done: - -```bash -uv init -``` - -Install: - -```bash -uv add dimos[dev,cpu,sim] -``` - -Test the Unitree Go2 robot in the simulator: - -```bash -uv run dimos-robot --simulation run unitree-g1 -``` - -Run your actual robot: - -```bash -uv run dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` - -### Without installing - -With `uv` you can run tools without having to explicitly install: - -```bash -uvx --from dimos dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` - -## With `pip` - -Create an environment if not already done: - -```bash -python -m venv .venv -. .venv/bin/activate -``` - -Install: - -```bash -pip install dimos[dev,cpu,sim] -``` - -Test the Unitree Go2 robot in the simulator: - -```bash -dimos-robot --simulation run unitree-g1 -``` - -Run your actual robot: - -```bash -dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` From 6d2924372063e360a6bec756a46313beb1f9284a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 21:13:50 -0800 Subject: [PATCH 014/136] consolidate and finish development docs --- README.md | 60 +++++----- docs/development.md | 277 +++++++++++++++++++++++++++++--------------- 2 files changed, 207 insertions(+), 130 deletions(-) diff --git a/README.md b/README.md index cc48e95135..13185f040f 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Dimensional is an open-source framework for adding customized general intelligen - Linux, tested on Ubuntu 22.04, 24.04 - MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* - - instead of the apt-get command below run: `xcode-select --install && brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` + - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` ```sh sudo apt-get update @@ -179,53 +179,45 @@ There are several tools: - Skills: Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools -# How do I Contribute / Hack on DimOS? +# Contributing / Building From Source -## Typical Installation +For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. -Very similar to a normal installation, here are the steps to get the repo and get it running for development. +If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development.md). + +## Quickstart Development (For Hacking, not Proper Development) ```bash + +# On Ubuntu 22.04 or 24.04 +if [ "$OSTYPE" = "linux-gnu" ]; then + sudo apt-get update + sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev +# On macOS (12.6 or newer) +elif [ "$(uname)" = "Darwin" ]; then + # install homebrew + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + # install dependencies + brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python +fi + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git cd dimos -sudo apt-get update -sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev + # install uv for python curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" + # create & activate a virtualenv (needed for dimos) uv venv && . .venv/bin/activate -# install dimos -uv pip install -e '.[base,dev,misc,unitree,drone]' -# test the install (takes about 3 minutes) -uv run pytest dimos -``` - - +``` Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: diff --git a/docs/development.md b/docs/development.md index 838bae6fdb..63dfb717b5 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,30 +1,103 @@ # Development Environment Guide -## Approach +1. Pick your setup (system install, dev container, docker, nix) +2. Best How to test/hack dimos (overview and our practices) +3. How to make a PR -We optimise for flexibility—if your favourite editor is **notepad.exe**, you’re good to go. Everything below is tooling for convenience. + + +# 1. Setup + +All the tools below are optional and for your convenience. If you can get the code running on temple OS with a package manager you wrote yourself, all the power to you. --- -## Dev Containers +## Setup Option A: System Install + +### Why pick this setup? (pros/cons/when-to-use) + +* Downside: not reliable, mutates your global system, causing (and receiving) side effects +* Upside: Often good for a quick hack or exploring +* Upside: Sometimes easier for CUDA/GPU acceleration +* Use when: you understand system package management (arch linux user) or you don't care about making changes to your system + +### How to setup DimOS + +```bash +# System dependencies + +# On Ubuntu 22.04 or 24.04 +if [ "$OSTYPE" = "linux-gnu" ]; then + sudo apt-get update + sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev pre-commit +# On macOS (12.6 or newer) +elif [ "$(uname)" = "Darwin" ]; then + # install homebrew + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + # install dependencies + brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python pre-commit +fi -Dev containers give us a reproducible, container-based workspace identical to CI. +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" -### Why use them? +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +cd dimos -* Consistent toolchain across all OSs. -* Unified formatting, linting and type-checking. -* Zero host-level dependencies (apart from Docker). -### IDE quick start +# create & activate a virtualenv (needed for dimos) +uv venv && . .venv/bin/activate -Install the *Dev Containers* plug-in for VS Code, Cursor, or your IDE of choice (you’ll likely be prompted automatically when you open our repo). +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' -### Shell only quick start +# setup pre-commit +pre-commit install + +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: + +- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector +- **contact_graspnet_pytorch**: Required for robotic grasp prediction + +You can install them with: + +```bash +uv add git+https://github.com/openai/CLIP.git +uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git +uv add git+https://github.com/facebookresearch/detectron2.git +``` + +## Setup Option B: Dev Containers (Recommended) + +### Why pick this setup? (pros/cons/when-to-use) + +* Upside: Reliable and consistent across OS's +* Upside: Unified formatting, linting and type-checking. +* Upside: Other than Docker, it won't touch your operating system (no side effects) +* Downside: It runs in a VM: slower, issues with GPU/CUDA, issues with hardware access like Webcam access, Networking, etc +* Upside: Your IDE-integrated vibe coding agent will "just work" +* Use when: You're not sure what option to pick + +### Quickstart + +First [install Docker](https://docs.docker.com/get-started/get-docker/) if you haven't already. + +Install the *Dev Containers* plug-in for VS Code, Cursor, or your IDE of choice. Clone the repo, open it in your IDE, and the IDE should prompt you to open in using a Dev Container. + +### Don't like IDE's? Use devcontainer CLI directly Terminal within your IDE should use devcontainer transparently given you installed the plugin, but in case you want to run our shell without an IDE, you can use `./bin/dev` (it depends on npm/node being installed) +
+Click to see how to use it in the command line + ```sh ./bin/dev devcontainer CLI (https://github.com/devcontainers/cli) not found. Install into repo root? (y/n): y @@ -56,125 +129,137 @@ The script will: You’ll land in the workspace as **root** with all project tooling available. -## Pre-Commit Hooks - -We use [pre-commit](https://pre-commit.com) (config in `.pre-commit-config.yaml`) to enforce formatting, licence headers, EOLs, LFS checks, etc. Hooks run in **milliseconds**. -Hooks also run in CI; any auto-fixes are committed back to your PR, so local installation is optional — but gives faster feedback. - -```sh -CRLF end-lines checker...................................................Passed -CRLF end-lines remover...................................................Passed -Insert license in comments...............................................Passed -ruff format..............................................................Passed -check for case conflicts.................................................Passed -check json...............................................................Passed -check toml...............................................................Passed -check yaml...............................................................Passed -format json..............................................................Passed -LFS data.................................................................Passed - -``` -Given your editor uses ruff via devcontainers (which it should) actual auto-commit hook won't ever reformat your code - IDE will have already done this. +
-### Running hooks manually +## Setup Option C: Nix Flake + direnv -Given your editor uses git via devcontainers (which it should) auto-commit hooks will run automatically, this is in case you want to run them manually. +### Why pick this setup? (pros/cons/when-to-use) -Inside the dev container (Your IDE will likely run this transparently for each commit if using devcontainer plugin): +* Upside: Faster and more reliable than Dev Containers (no emulation) +* Upside: Nearly as isolated as Docker, but has full hardware access (CUDA, Webcam, networking) +* Downside: Not hard, but you need to install/understand [direnv](https://direnv.net/) (which you probably should do anyway) +* Downside: Nix is not user-friendly (IDE integration is not as good as Dev Containers) +* Use when: you need reliability and don't mind a one-time startup delay -```sh -pre-commit run --all-files -``` +### Quickstart -### Installing pre-commit on your host +Install and activate [direnv](https://direnv.net/). ```sh -apt install pre-commit # or brew install pre-commit -pre-commit install # install git hook -pre-commit run --all-files +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix .envrc +cp .envrc.nix .envrc +# this is going to take a while +direnv allow +direnv reload +direnv status + +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos ``` +## Setup Option D: Nix Flake - Isolated/Reliable ---- +### Why pick this setup? (pros/cons/when-to-use) -## Testing +* Use when: you need absolute reliability (use this if you want it to work first try) and don't mind a startup delay +* Upside: Doesn't need direnv, and has most of the other benefits of Nix +* Downside: Have to manually enter the environment (like `./venv/bin/activate` but slower) +* Upside: If you're you're using a basic shell, you'll get a nicely customized shell +* Downside: If you have hyper-customized your shell (fish, riced zsh, etc), you'll have to deal with someone else's preferences +* Downside: Your vibe coding agent will basically be unable to run tests for you (they don't understand how to enter the environment) -All tests run with **pytest** inside the dev container, ensuring local results match CI. +### Quickstart -### Basic usage +Install and activate [direnv](https://direnv.net/). ```sh -./bin/dev # start container -pytest # run all tests beneath the current directory +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix development shell +nix develop '.#isolated' ``` -Depending on which dir you are in, only tests from that dir will run, which is convinient when developing - you can frequently validate your feature tree. - -Your vibe coding agent will know to use these tests via the devcontainer so it can validate it's work. - - -#### Useful options - -| Purpose | Command | -| -------------------------- | ----------------------- | -| Show `print()` output | `pytest -s` | -| Filter by name substring | `pytest -k ""` | -| Run tests with a given tag | `pytest -m ` | - - -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) - -You can enable a tag by selecting -m - these are configured in `./pyproject.toml` +Once inside the shell, run: ```sh -root@dimos:/workspaces/dimos/dimos # pytest -sm vis -k my_visualization -... +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos ``` -Classic development run within a subtree: +
+
-```sh -./bin/dev - -... container init ... +# 2. How to Test and Modify DimOS -root@dimos:/workspaces/dimos # cd dimos/robot/unitree_webrtc/ -root@dimos:/workspaces/dimos/dimos/robot/unitree_webrtc # pytest -collected 27 items / 22 deselected / 5 selected +## Where is `` located? -type/test_map.py::test_robot_mapping PASSED -type/test_timeseries.py::test_repr PASSED -type/test_timeseries.py::test_equals PASSED -type/test_timeseries.py::test_range PASSED -type/test_timeseries.py::test_duration PASSED +* `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. +* `dimos/robot/`: Robot-specific modules live here. +* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. +* `dimos/dashboard/`: Contains code related to visualization. +* `dimos/protocol/`: Defines low level stuff for communication between modules. -``` +## Testing -Showing prints: +We use both pytest and manual testing. ```sh -root@dimos:/workspaces/dimos/dimos/robot/unitree_webrtc/type # pytest -s test_odometry.py -test_odometry.py::test_odometry_conversion_and_count Odom ts(2025-05-30 13:52:03) pos(→ Vector Vector([0.432199 0.108042 0.316589])), rot(↑ Vector Vector([ 7.7200000e-04 -9.1280000e-03 3.006 -8621e+00])) yaw(172.3°) -Odom ts(2025-05-30 13:52:03) pos(→ Vector Vector([0.433629 0.105965 0.316143])), rot(↑ Vector Vector([ 0.003814 -0.006436 2.99591235])) yaw(171.7°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.434459 0.104739 0.314794])), rot(↗ Vector Vector([ 0.005558 -0.004183 3.00068456])) yaw(171.9°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.435621 0.101699 0.315852])), rot(↑ Vector Vector([ 0.005391 -0.006002 3.00246893])) yaw(172.0°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.436457 0.09857 0.315254])), rot(↑ Vector Vector([ 0.003358 -0.006916 3.00347172])) yaw(172.1°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.435535 0.097022 0.314399])), rot(↑ Vector Vector([ 1.88300000e-03 -8.17800000e-03 3.00573432e+00])) yaw(172.2°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.433739 0.097553 0.313479])), rot(↑ Vector Vector([ 8.10000000e-05 -8.71700000e-03 3.00729616e+00])) yaw(172.3°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.430924 0.09859 0.31322 ])), rot(↑ Vector Vector([ 1.84000000e-04 -9.68700000e-03 3.00945623e+00])) yaw(172.4°) -... etc +pytest # run all tests at or below the current directory ``` ---- -## Cheatsheet +### Testing Cheatsheet | Action | Command | | --------------------------- | ---------------------------- | -| Enter dev container | `./bin/dev` | -| Run all pre-commit hooks | `pre-commit run --all-files` | -| Install hooks in local repo | `pre-commit install` | | Run tests in current path | `pytest` | | Filter tests by name | `pytest -k ""` | | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | +| Run all pre-commit hooks | `pre-commit run --all-files` | + +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) + +You can enable a tag by selecting -m - these are configured in `./pyproject.toml` + + +# 3. How to Make a PR +- Open the PR against the `dev` branch (not `main`) +- **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) +- Less changed files = better +- If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. +- We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). +- Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. +- So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. +- Don't commit large (>500kb) files, instead, use git lfs to store them (see `git/config` for how to set this up) +- If you're a new hire at DimOS: + - Smaller PR's are better. + - Only open a PR when you're okay with us spending AI tokens reviewing it (don't just open a trash PR and then fix it, wait till the code is mostly done) + - If there are 3 highly-intertwined bugs, make 3 PRs, not 1 PR. Yes it is more dev work, but review time is the bottleneck (not dev time). One line PR's are the easiest thing to review. + - When the AI (currently Greptile) comments on the code, respond. Sometimes Greptile is dumb as rocks but, as a reviewer, it's nice to see a finished conversation. + - Did we mention smaller PR's are better? From 62e753776d28b72c13777d78a65092f65c3ad5e3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 21:27:33 -0800 Subject: [PATCH 015/136] remove redundant doc --- docs/old/running_without_devcontainer.md | 21 --------------------- 1 file changed, 21 deletions(-) delete mode 100644 docs/old/running_without_devcontainer.md diff --git a/docs/old/running_without_devcontainer.md b/docs/old/running_without_devcontainer.md deleted file mode 100644 index d06785e359..0000000000 --- a/docs/old/running_without_devcontainer.md +++ /dev/null @@ -1,21 +0,0 @@ -install nix, - -https://nixos.wiki/wiki/Nix_Installation_Guide -```sh -sudo install -d -m755 -o $(id -u) -g $(id -g) /nix -curl -L https://nixos.org/nix/install | sh -``` - -install direnv -https://direnv.net/ -```sh -apt-get install direnv -echo 'eval "$(direnv hook bash)"' >> ~/.bashrc -``` - -allow direnv in dimos will take a bit to pull the packages, -from that point on your env is standardized -```sh -cd dimos -direnv allow -``` From 10484d5466f4fb7db5cdc29deccb80cd5161807f Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 22:01:11 -0800 Subject: [PATCH 016/136] docs reorganization pt1 --- README.md | 2 +- dimos/utils/docs/doclinks.md | 96 ------ docs/agents/docs/doclinks.md | 21 -- docs/agents/docs/index.md | 192 ----------- .../assets/codeblocks_example.svg | 0 .../assets/pikchr_basic.svg | 0 .../assets/pikchr_branch.svg | 0 .../assets/pikchr_explicit.svg | 0 .../assets/pikchr_labels.svg | 0 .../assets/pikchr_sizing.svg | 0 .../large_file_management.md} | 0 docs/{development.md => development/main.md} | 3 +- .../writing_docs.md} | 312 ++++++++++++++---- docs/old/ci.md | 146 -------- docs/old/jetson.MD | 72 ---- docs/old/modules.md | 165 --------- docs/old/modules_CN.md | 188 ----------- docs/old/ros_navigation.md | 284 ---------------- docs/old/testing_stream_reply.md | 174 ---------- 19 files changed, 243 insertions(+), 1412 deletions(-) delete mode 100644 dimos/utils/docs/doclinks.md delete mode 100644 docs/agents/docs/doclinks.md delete mode 100644 docs/agents/docs/index.md rename docs/{agents/docs => development}/assets/codeblocks_example.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_basic.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_branch.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_explicit.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_labels.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_sizing.svg (100%) rename docs/{data.md => development/large_file_management.md} (100%) rename docs/{development.md => development/main.md} (97%) rename docs/{agents/docs/codeblocks.md => development/writing_docs.md} (55%) delete mode 100644 docs/old/ci.md delete mode 100644 docs/old/jetson.MD delete mode 100644 docs/old/modules.md delete mode 100644 docs/old/modules_CN.md delete mode 100644 docs/old/ros_navigation.md delete mode 100644 docs/old/testing_stream_reply.md diff --git a/README.md b/README.md index 13185f040f..620d76469a 100644 --- a/README.md +++ b/README.md @@ -183,7 +183,7 @@ There are several tools: For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. -If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development.md). +If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/main.md). ## Quickstart Development (For Hacking, not Proper Development) diff --git a/dimos/utils/docs/doclinks.md b/dimos/utils/docs/doclinks.md deleted file mode 100644 index dce2e67fec..0000000000 --- a/dimos/utils/docs/doclinks.md +++ /dev/null @@ -1,96 +0,0 @@ -# doclinks - -A Markdown link resolver that automatically fills in correct file paths for code references in documentation. - -## What it does - -When writing docs, you can use placeholder links like: - - -```markdown -See [`service/spec.py`]() for the implementation. -``` - - -Running `doclinks` resolves these to actual paths: - - -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - - -## Features - - -- **Code file links**: `[`filename.py`]()` resolves to the file's path -- **Symbol line linking**: If another backticked term appears on the same line, it finds that symbol in the file and adds `#L`: - ```markdown - See `Configurable` in [`config.py`]() - → [`config.py`](/path/config.py#L42) - ``` -- **Doc-to-doc links**: `[Modules](.md)` resolves to `modules.md` or `modules/index.md` - -- **Multiple link modes**: absolute, relative, or GitHub URLs -- **Watch mode**: Automatically re-process on file changes -- **Ignore regions**: Skip sections with `` comments - -## Usage - -```bash -# Process a single file -doclinks docs/guide.md - -# Process a directory recursively -doclinks docs/ - -# Relative links (from doc location) -doclinks --link-mode relative docs/ - -# GitHub links -doclinks --link-mode github \ - --github-url https://github.com/org/repo docs/ - -# Dry run (preview changes) -doclinks --dry-run docs/ - -# CI check (exit 1 if changes needed) -doclinks --check docs/ - -# Watch mode (auto-update on changes) -doclinks --watch docs/ -``` - -## Options - -| Option | Description | -|--------------------|-------------------------------------------------| -| `--root PATH` | Repository root (default: auto-detect git root) | -| `--link-mode MODE` | `absolute` (default), `relative`, or `github` | -| `--github-url URL` | Base GitHub URL (required for github mode) | -| `--github-ref REF` | Branch/ref for GitHub links (default: `main`) | -| `--dry-run` | Show changes without modifying files | -| `--check` | Exit with error if changes needed (for CI) | -| `--watch` | Watch for changes and re-process | - -## Link patterns - - -| Pattern | Description | -|----------------------|------------------------------------------------| -| `[`file.py`]()` | Code file reference (empty or any link) | -| `[`path/file.py`]()` | Code file with partial path for disambiguation | -| `[`file.py`](#L42)` | Preserves existing line fragments | -| `[Doc Name](.md)` | Doc-to-doc link (resolves by name) | - - -## How resolution works - -The tool builds an index of all files in the repo. For `/dimos/protocol/service/spec.py`, it creates lookup entries for: - -- `spec.py` -- `service/spec.py` -- `protocol/service/spec.py` -- `dimos/protocol/service/spec.py` - -Use longer paths when multiple files share the same name. diff --git a/docs/agents/docs/doclinks.md b/docs/agents/docs/doclinks.md deleted file mode 100644 index d5533c5983..0000000000 --- a/docs/agents/docs/doclinks.md +++ /dev/null @@ -1,21 +0,0 @@ -When writing or editing markdown documentation, use `doclinks` tool to resolve file references. - -Full documentation if needed: [`utils/docs/doclinks.md`](/dimos/utils/docs/doclinks.md) - -## Syntax - - -| Pattern | Example | -|-------------|-----------------------------------------------------| -| Code file | `[`service/spec.py`]()` → resolves path | -| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | -| Doc link | `[Configuration](.md)` → resolves to doc | - - -## Usage - -```bash -doclinks docs/guide.md # single file -doclinks docs/ # directory -doclinks --dry-run ... # preview only -``` diff --git a/docs/agents/docs/index.md b/docs/agents/docs/index.md deleted file mode 100644 index bec2ce79e6..0000000000 --- a/docs/agents/docs/index.md +++ /dev/null @@ -1,192 +0,0 @@ - -# Code Blocks - -**All code blocks must be executable.** -Never write illustrative/pseudo code blocks. -If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. - -After writing a code block in your markdown file, you can run it by executing -`md-babel-py run document.md` - -more information on this tool is in [codeblocks](/docs/agents/docs_agent/codeblocks.md) - - -# Code or Docs Links - -After adding a link to a doc run - -`doclinks document.md` - -### Code file references -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -After running doclinks, becomes: -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -### Symbol auto-linking -Mention a symbol on the same line to auto-link to its line number: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` - -Becomes: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` -### Doc-to-doc references -Use `.md` as the link target: -```markdown -See [Configuration](/docs/api/configuration.md) for more details. -``` - -Becomes: -```markdown -See [Configuration](/docs/concepts/configuration.md) for more details. -``` - -More information on this in [doclinks](/docs/agents/docs_agent/doclinks.md) - - -# Pikchr - -[Pikchr](https://pikchr.org/) is a diagram language from SQLite. Use it for flowcharts and architecture diagrams. - -**Important:** Always wrap pikchr blocks in `
` tags so the source is collapsed by default on GitHub. The rendered SVG stays visible outside the fold. Code blocks (Python, etc.) should NOT be folded—they're meant to be read. - -## Basic syntax - -
-diagram source - -```pikchr fold output=assets/pikchr_basic.svg -color = white -fill = none - -A: box "Step 1" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box "Step 2" rad 5px fit wid 170% ht 170% -arrow right 0.3in -C: box "Step 3" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_basic.svg) - -## Box sizing - -Use `fit` with percentage scaling to auto-size boxes with padding: - -
-diagram source - -```pikchr fold output=assets/pikchr_sizing.svg -color = white -fill = none - -# fit wid 170% ht 170% = auto-size + padding -A: box "short" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box ".subscribe()" rad 5px fit wid 170% ht 170% -arrow right 0.3in -C: box "two lines" "of text" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_sizing.svg) - -The pattern `fit wid 170% ht 170%` means: auto-size to text, then scale width by 170% and height by 170%. - -For explicit sizing (when you need consistent box sizes): - -
-diagram source - -```pikchr fold output=assets/pikchr_explicit.svg -color = white -fill = none - -A: box "Step 1" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box "Step 2" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_explicit.svg) - -## Common settings - -Always start with: - -``` -color = white # text color -fill = none # transparent box fill -``` - -## Branching paths - -
-diagram source - -```pikchr fold output=assets/pikchr_branch.svg -color = white -fill = none - -A: box "Input" rad 5px fit wid 170% ht 170% -arrow -B: box "Process" rad 5px fit wid 170% ht 170% - -# Branch up -arrow from B.e right 0.3in then up 0.35in then right 0.3in -C: box "Path A" rad 5px fit wid 170% ht 170% - -# Branch down -arrow from B.e right 0.3in then down 0.35in then right 0.3in -D: box "Path B" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_branch.svg) - -**Tip:** For tree/hierarchy diagrams, prefer left-to-right layout (root on left, children branching right). This reads more naturally and avoids awkward vertical stacking. - -## Adding labels - -
-diagram source - -```pikchr fold output=assets/pikchr_labels.svg -color = white -fill = none - -A: box "Box" rad 5px fit wid 170% ht 170% -text "label below" at (A.x, A.y - 0.4in) -``` - -
- - -![output](assets/pikchr_labels.svg) - -## Reference - -| Element | Syntax | -|---------|--------| -| Box | `box "text" rad 5px wid Xin ht Yin` | -| Arrow | `arrow right 0.3in` | -| Oval | `oval "text" wid Xin ht Yin` | -| Text | `text "label" at (X, Y)` | -| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | - -See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/agents/docs/assets/codeblocks_example.svg b/docs/development/assets/codeblocks_example.svg similarity index 100% rename from docs/agents/docs/assets/codeblocks_example.svg rename to docs/development/assets/codeblocks_example.svg diff --git a/docs/agents/docs/assets/pikchr_basic.svg b/docs/development/assets/pikchr_basic.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_basic.svg rename to docs/development/assets/pikchr_basic.svg diff --git a/docs/agents/docs/assets/pikchr_branch.svg b/docs/development/assets/pikchr_branch.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_branch.svg rename to docs/development/assets/pikchr_branch.svg diff --git a/docs/agents/docs/assets/pikchr_explicit.svg b/docs/development/assets/pikchr_explicit.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_explicit.svg rename to docs/development/assets/pikchr_explicit.svg diff --git a/docs/agents/docs/assets/pikchr_labels.svg b/docs/development/assets/pikchr_labels.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_labels.svg rename to docs/development/assets/pikchr_labels.svg diff --git a/docs/agents/docs/assets/pikchr_sizing.svg b/docs/development/assets/pikchr_sizing.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_sizing.svg rename to docs/development/assets/pikchr_sizing.svg diff --git a/docs/data.md b/docs/development/large_file_management.md similarity index 100% rename from docs/data.md rename to docs/development/large_file_management.md diff --git a/docs/development.md b/docs/development/main.md similarity index 97% rename from docs/development.md rename to docs/development/main.md index 63dfb717b5..050982cc57 100644 --- a/docs/development.md +++ b/docs/development/main.md @@ -251,12 +251,13 @@ You can enable a tag by selecting -m - these are configured in `./pyp # 3. How to Make a PR - Open the PR against the `dev` branch (not `main`) - **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) +- If you're writing documentation, see [writing_docs.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/writing_docs.md) for how to write code blocks. - Less changed files = better - If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. +- If you have large (>500kb) files, see [large_files.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/large_files.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. -- Don't commit large (>500kb) files, instead, use git lfs to store them (see `git/config` for how to set this up) - If you're a new hire at DimOS: - Smaller PR's are better. - Only open a PR when you're okay with us spending AI tokens reviewing it (don't just open a trash PR and then fix it, wait till the code is mostly done) diff --git a/docs/agents/docs/codeblocks.md b/docs/development/writing_docs.md similarity index 55% rename from docs/agents/docs/codeblocks.md rename to docs/development/writing_docs.md index 323f1c0c50..6a73d0acf8 100644 --- a/docs/agents/docs/codeblocks.md +++ b/docs/development/writing_docs.md @@ -1,10 +1,226 @@ -# Executable Code Blocks +# Need-to-know Things + +1. How we resolve file references (linking) +2. How we make of our code blocks executable and test them. +3. How we make diagrams + + +
+
+ +# 1. Use Doclinks to Resolve file references + +## Syntax + + +| Pattern | Example | +|-------------|-----------------------------------------------------| +| Code file | `[`service/spec.py`]()` → resolves path | +| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | +| Doc link | `[Configuration](.md)` → resolves to doc | + + +## Usage + +```bash +doclinks docs/guide.md # single file +doclinks docs/ # directory +doclinks --dry-run ... # preview only +``` + +## Full Documentation + +
+Click to see full documentation + +## What it does + +When writing docs, you can use placeholder links like: + + +```markdown +See [`service/spec.py`]() for the implementation. +``` + + +Running `doclinks` resolves these to actual paths: + + +```markdown +See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. +``` + + +## Features + + +- **Code file links**: `[`filename.py`]()` resolves to the file's path +- **Symbol line linking**: If another backticked term appears on the same line, it finds that symbol in the file and adds `#L`: + ```markdown + See `Configurable` in [`config.py`]() + → [`config.py`](/path/config.py#L42) + ``` +- **Doc-to-doc links**: `[Modules](.md)` resolves to `modules.md` or `modules/index.md` + +- **Multiple link modes**: absolute, relative, or GitHub URLs +- **Watch mode**: Automatically re-process on file changes +- **Ignore regions**: Skip sections with `` comments + +## Usage + +```bash +# Process a single file +doclinks docs/guide.md + +# Process a directory recursively +doclinks docs/ + +# Relative links (from doc location) +doclinks --link-mode relative docs/ + +# GitHub links +doclinks --link-mode github \ + --github-url https://github.com/org/repo docs/ + +# Dry run (preview changes) +doclinks --dry-run docs/ + +# CI check (exit 1 if changes needed) +doclinks --check docs/ + +# Watch mode (auto-update on changes) +doclinks --watch docs/ +``` + +## Options + +| Option | Description | +|--------------------|-------------------------------------------------| +| `--root PATH` | Repository root (default: auto-detect git root) | +| `--link-mode MODE` | `absolute` (default), `relative`, or `github` | +| `--github-url URL` | Base GitHub URL (required for github mode) | +| `--github-ref REF` | Branch/ref for GitHub links (default: `main`) | +| `--dry-run` | Show changes without modifying files | +| `--check` | Exit with error if changes needed (for CI) | +| `--watch` | Watch for changes and re-process | + +## Link patterns + + +| Pattern | Description | +|----------------------|------------------------------------------------| +| `[`file.py`]()` | Code file reference (empty or any link) | +| `[`path/file.py`]()` | Code file with partial path for disambiguation | +| `[`file.py`](#L42)` | Preserves existing line fragments | +| `[Doc Name](.md)` | Doc-to-doc link (resolves by name) | + + +## How resolution works + +The tool builds an index of all files in the repo. For `/dimos/protocol/service/spec.py`, it creates lookup entries for: + +- `spec.py` +- `service/spec.py` +- `protocol/service/spec.py` +- `dimos/protocol/service/spec.py` + +Use longer paths when multiple files share the same name. + +
+ + + +
+
+ +# 2. Code Blocks Must Be Executable We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. ## Golden Rule -**All code blocks must be executable.** Never write illustrative/pseudo code blocks. If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. +**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. + +## Installation + +
+Click to see full installation instructions + +### Nix (recommended) + +```sh skip +# (assuming you have nix) + +# Run directly from GitHub +nix run github:leshy/md-babel-py -- run README.md --stdout + +# run locally +nix run . -- run README.md --stdout +``` + +### Docker + +```sh skip +# Pull from Docker Hub +docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout + +# Or build locally via Nix +nix build .#docker # builds tarball to ./result +docker load < result # loads image from tarball +docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout +``` + +### pipx + +```sh skip +pipx install md-babel-py +# or: uv pip install md-babel-py +md-babel-py run README.md --stdout +``` + +If not using nix or docker, evaluators require system dependencies: + +| Language | System packages | +|-----------|-----------------------------| +| python | python3 | +| node | nodejs | +| dot | graphviz | +| asymptote | asymptote, texlive, dvisvgm | +| pikchr | pikchr | +| openscad | openscad, xvfb, imagemagick | +| diagon | diagon | + +```sh skip +# Arch Linux +sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick + +# Debian/Ubuntu +sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad +``` + +Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. + +## Usage + +```sh skip +# Edit file in-place +md-babel-py run document.md + +# Output to separate file +md-babel-py run document.md --output result.md + +# Print to stdout +md-babel-py run document.md --stdout + +# Only run specific languages +md-babel-py run document.md --lang python,sh + +# Dry run - show what would execute +md-babel-py run document.md --dry-run +``` + +
+ ## Running @@ -113,6 +329,19 @@ plt.savefig('{output}', transparent=True) ![output](assets/matplotlib-demo.svg) + + + + + +
+
+ +# 3. Diagrams + +We have many diagraming tools. View source code of this page to see examples. + + ### Pikchr SQLite's diagram language: @@ -240,75 +469,14 @@ A -> C └───┘ ``` -## Install +## Reference -### Nix (recommended) +| Element | Syntax | +|---------|--------| +| Box | `box "text" rad 5px wid Xin ht Yin` | +| Arrow | `arrow right 0.3in` | +| Oval | `oval "text" wid Xin ht Yin` | +| Text | `text "label" at (X, Y)` | +| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | -```sh skip -# Run directly from GitHub -nix run github:leshy/md-babel-py -- run README.md --stdout - -# Or clone and run locally -nix run . -- run README.md --stdout -``` - -### Docker - -```sh skip -# Pull from Docker Hub -docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout - -# Or build locally via Nix -nix build .#docker # builds tarball to ./result -docker load < result # loads image from tarball -docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout -``` - -### pipx - -```sh skip -pipx install md-babel-py -# or: uv pip install md-babel-py -md-babel-py run README.md --stdout -``` - -If not using nix or docker, evaluators require system dependencies: - -| Language | System packages | -|-----------|-----------------------------| -| python | python3 | -| node | nodejs | -| dot | graphviz | -| asymptote | asymptote, texlive, dvisvgm | -| pikchr | pikchr | -| openscad | openscad, xvfb, imagemagick | -| diagon | diagon | - -```sh skip -# Arch Linux -sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick - -# Debian/Ubuntu -sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad -``` - -Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. - -## Usage - -```sh skip -# Edit file in-place -md-babel-py run document.md - -# Output to separate file -md-babel-py run document.md --output result.md - -# Print to stdout -md-babel-py run document.md --stdout - -# Only run specific languages -md-babel-py run document.md --lang python,sh - -# Dry run - show what would execute -md-babel-py run document.md --dry-run -``` +See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/old/ci.md b/docs/old/ci.md deleted file mode 100644 index ac9b11115a..0000000000 --- a/docs/old/ci.md +++ /dev/null @@ -1,146 +0,0 @@ -# Continuous Integration Guide - -> *If you are ******not****** editing CI-related files, you can safely ignore this document.* - -Our GitHub Actions pipeline lives in **`.github/workflows/`** and is split into three top-level workflows: - -| Workflow | File | Purpose | -| ----------- | ------------- | -------------------------------------------------------------------- | -| **cleanup** | `cleanup.yml` | Auto-formats code with *pre-commit* and pushes fixes to your branch. | -| **docker** | `docker.yml` | Builds (and caches) our Docker image hierarchy. | -| **tests** | `tests.yml` | Pulls the *dev* image and runs the test suite. | - ---- - -## `cleanup.yml` - -* Checks out the branch. -* Executes **pre-commit** hooks. -* If hooks modify files, commits and pushes the changes back to the same branch. - -> This guarantees consistent formatting even if the developer has not installed pre-commit locally. - ---- - -## `tests.yml` - -* Pulls the pre-built **dev** container image. -* Executes: - -```bash -pytest -``` - -That’s it—making the job trivial to reproduce locally via: - -```bash -./bin/dev # enter container -pytest # run tests -``` - ---- - -## `docker.yml` - -### Objectives - -1. **Layered images**: each image builds on its parent, enabling parallel builds once dependencies are ready. -2. **Speed**: build children as soon as parents finish; leverage aggressive caching. -3. **Minimal work**: skip images whose context hasn’t changed. - -### Current hierarchy - - -``` - ┌──────┐ - │ubuntu│ - └┬────┬┘ - ┌▽──┐┌▽───────┐ - │ros││python │ - └┬──┘└───────┬┘ - ┌▽─────────┐┌▽──┐ - │ros-python││dev│ - └┬─────────┘└───┘ - ┌▽──────┐ - │ros-dev│ - └───────┘ -``` - -* ghcr.io/dimensionalos/ros:dev -* ghcr.io/dimensionalos/python:dev -* ghcr.io/dimensionalos/ros-python:dev -* ghcr.io/dimensionalos/ros-dev:dev -* ghcr.io/dimensionalos/dev:dev - -> **Note**: The diagram shows only currently active images; the system is extensible—new combinations are possible, builds can be run per branch and as parallel as possible - - -``` - ┌──────┐ - │ubuntu│ - └┬────┬┘ - ┌▽──┐┌▽────────────────────────┐ - │ros││python │ - └┬──┘└───────────────────┬────┬┘ - ┌▽─────────────────────┐┌▽──┐┌▽──────┐ - │ros-python ││dev││unitree│ - └┬────────┬───────────┬┘└───┘└───────┘ - ┌▽──────┐┌▽─────────┐┌▽──────────┐ - │ros-dev││ros-jetson││ros-unitree│ - └───────┘└──────────┘└───────────┘ -``` - -### Branch-aware tagging - -When a branch triggers a build: - -* Only images whose context changed are rebuilt. -* New images receive the tag `:`. -* Unchanged parents are pulled from the registry, e.g. - -given we made python requirements.txt changes, but no ros changes, image dep graph would look like this: - -``` -ghcr.io/dimensionalos/ros:dev → ghcr.io/dimensionalos/ros-python:my_branch → ghcr.io/dimensionalos/dev:my_branch -``` - -### Job matrix & the **check-changes** step - -To decide what to build we run a `check-changes` job that compares the diff against path filters: - -```yaml -filters: | - ros: - - .github/workflows/_docker-build-template.yml - - .github/workflows/docker.yml - - docker/base-ros/** - - python: - - docker/base-python/** - - requirements*.txt - - dev: - - docker/dev/** -``` - -This populates a build matrix (ros, python, dev) with `true/false` flags. - -### The dependency execution issue - -Ideally a child job (e.g. **ros-python**) should depend on both: - -* **check-changes** (to know if it *should* run) -* Its **parent image job** (to wait for the artifact) - -GitHub Actions can’t express “run only if *both* conditions are true *and* the parent job wasn’t skipped”. - -We are using `needs: [check-changes, ros]` to ensure the job runs after the ros build, but if ros build has been skipped we need `if: always()` to ensure that the build runs anyway. -Adding `always` for some reason completely breaks the conditional check, we cannot have OR, AND operators, it just makes the job _always_ run, which means we build python even if we don't need to. - -This is unfortunate as the build takes ~30 min first time (a few minutes afterwards thanks to caching) and I've spent a lot of time on this, lots of viable seeming options didn't pan out and probably we need to completely rewrite and own the actions runner and not depend on github structure at all. Single job called `CI` or something, within our custom docker image. - ---- - -## `run-tests` (job inside `docker.yml`) - -After all requested images are built, this job triggers **tests.yml**, passing the freshly created *dev* image tag so the suite runs against the branch-specific environment. diff --git a/docs/old/jetson.MD b/docs/old/jetson.MD deleted file mode 100644 index a4d06e3255..0000000000 --- a/docs/old/jetson.MD +++ /dev/null @@ -1,72 +0,0 @@ -# DimOS Jetson Setup Instructions -Tested on Jetpack 6.2, CUDA 12.6 - -## Required system dependencies -`sudo apt install portaudio19-dev python3-pyaudio` - -## Installing cuSPARSELt -https://ninjalabo.ai/blogs/jetson_pytorch.html - -```bash -wget https://developer.download.nvidia.com/compute/cusparselt/0.7.0/local_installers/cusparselt-local-tegra-repo-ubuntu2204-0.7.0_1.0-1_arm64.deb -sudo dpkg -i cusparselt-local-tegra-repo-ubuntu2204-0.7.0_1.0-1_arm64.deb -sudo cp /var/cusparselt-local-tegra-repo-ubuntu2204-0.7.0/cusparselt-*-keyring.gpg /usr/share/keyrings/ -sudo apt-get update -sudo apt-get install libcusparselt0 libcusparselt-dev -ldconfig -``` -## Install Torch and Torchvision wheels - -Enter virtualenv -```bash -python3 -m venv venv -source venv/bin/activate -``` - -Wheels for jp6/cu126 -https://pypi.jetson-ai-lab.io/jp6/cu126 - -Check compatibility: -https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform-release-notes/pytorch-jetson-rel.html - -### Working torch wheel tested on Jetpack 6.2, CUDA 12.6 -`pip install --no-cache https://developer.download.nvidia.com/compute/redist/jp/v61/pytorch/torch-2.5.0a0+872d972e41.nv24.08.17622132-cp310-cp310-linux_aarch64.whl` - -### Install torchvision from source: -```bash -# Set version by checking above torchvision<-->torch compatibility - -# We use 0.20.0 -export VERSION=20 - -sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libopenblas-dev libavcodec-dev libavformat-dev libswscale-dev -git clone --branch release/0.$VERSION https://github.com/pytorch/vision torchvision -cd torchvision -export BUILD_VERSION=0.$VERSION.0 -python3 setup.py install --user # remove --user if installing in virtualenv -``` - -### Verify success: -```bash -$ python3 -import torch -print(torch.__version__) -print('CUDA available: ' + str(torch.cuda.is_available())) # Should be True -print('cuDNN version: ' + str(torch.backends.cudnn.version())) -a = torch.cuda.FloatTensor(2).zero_() -print('Tensor a = ' + str(a)) -b = torch.randn(2).cuda() -print('Tensor b = ' + str(b)) -c = a + b -print('Tensor c = ' + str(c)) - -$ python3 -import torchvision -print(torchvision.__version__) -``` - -## Install Onnxruntime-gpu - -Find pre-build wheels here for your specific JP/CUDA version: https://pypi.jetson-ai-lab.io/jp6 - -`pip install https://pypi.jetson-ai-lab.io/jp6/cu126/+f/4eb/e6a8902dc7708/onnxruntime_gpu-1.23.0-cp310-cp310-linux_aarch64.whl#sha256=4ebe6a8902dc7708434b2e1541b3fe629ebf434e16ab5537d1d6a622b42c622b` diff --git a/docs/old/modules.md b/docs/old/modules.md deleted file mode 100644 index 9cdbf586ac..0000000000 --- a/docs/old/modules.md +++ /dev/null @@ -1,165 +0,0 @@ -# Dimensional Modules - -The DimOS Module system enables distributed, multiprocess robotics applications using Dask for compute distribution and LCM (Lightweight Communications and Marshalling) for high-performance IPC. - -## Core Concepts - -### 1. Module Definition -Modules are Python classes that inherit from `dimos.core.Module` and define inputs, outputs, and RPC methods: - -```python -from dimos.core import Module, In, Out, rpc -from dimos.msgs.geometry_msgs import Vector3 - -class MyModule(Module): - # Declare inputs/outputs as class attributes initialized to None - data_in: In[Vector3] = None - data_out: Out[Vector3] = None - - def __init__(): - # Call parent Module init - super().__init__() - - @rpc - def remote_method(self, param): - """Methods decorated with @rpc can be called remotely""" - return param * 2 -``` - -### 2. Module Deployment -Modules are deployed across Dask workers using the `dimos.deploy()` method: - -```python -from dimos import core - -# Start Dask cluster with N workers -dimos = core.start(4) - -# Deploying modules allows for passing initialization parameters. -# In this case param1 and param2 are passed into Module init -module = dimos.deploy(Module, param1="value1", param2=123) -``` - -### 3. Stream Connections -Modules communicate via reactive streams using LCM transport: - -```python -# Configure LCM transport for outputs -module1.data_out.transport = core.LCMTransport("/topic_name", MessageType) - -# Connect module inputs to outputs -module2.data_in.connect(module1.data_out) - -# Access the underlying Observable stream -stream = module1.data_out.observable() -stream.subscribe(lambda msg: print(f"Received: {msg}")) -``` - -### 4. Module Lifecycle -```python -# Start modules to begin processing -module.start() # Calls the @rpc start() method if defined - -# Inspect module I/O configuration -print(module.io().result()) # Shows inputs, outputs, and RPC methods - -# Clean shutdown -dimos.shutdown() -``` - -## Real-World Example: Robot Control System - -```python -# Connection module wraps robot hardware/simulation -connection = dimos.deploy(ConnectionModule, ip=robot_ip) -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) -connection.video.transport = core.LCMTransport("/video", Image) - -# Perception module processes sensor data -perception = dimos.deploy(PersonTrackingStream, camera_intrinsics=[...]) -perception.video.connect(connection.video) -perception.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# Start processing -connection.start() -perception.start() - -# Enable tracking via RPC -perception.enable_tracking() - -# Get latest tracking data -data = perception.get_tracking_data() -``` - -## LCM Transport Configuration - -```python -# Standard LCM transport for simple types like lidar -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) - -# Pickle-based transport for complex Python objects / dictionaries -connection.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# Auto-configure LCM system buffers (required in containers) -from dimos.protocol import pubsub -pubsub.lcm.autoconf() -``` - -This architecture enables building complex robotic systems as composable, distributed modules that communicate efficiently via streams and RPC, scaling from single machines to clusters. - -# Dimensional Install -## Python Installation (Ubuntu 22.04) - -```bash -sudo apt install python3-venv - -# Clone the repository (dev branch, no submodules) -git clone -b dev https://github.com/dimensionalOS/dimos.git -cd dimos - -# Create and activate virtual environment -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# Install torch and torchvision if not already installed -# Example CUDA 11.7, Pytorch 2.0.1 (replace with your required pytorch version if different) -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -``` - -### Install dependencies -```bash -# CPU only (reccomended to attempt first) -pip install .[cpu,dev] - -# CUDA install -pip install .[cuda,dev] - -# Copy and configure environment variables -cp default.env .env -``` - -### Test install -```bash -# Run standard tests -pytest -s dimos/ - -# Test modules functionality -pytest -s -m module dimos/ - -# Test LCM communication -pytest -s -m lcm dimos/ -``` - -# Unitree Go2 Quickstart - -To quickly test the modules system, you can run the Unitree Go2 multiprocess example directly: - -```bash -# Make sure you have the required environment variables set -export ROBOT_IP= - -# Run the multiprocess Unitree Go2 example -python dimos/robot/unitree_webrtc/multiprocess/unitree_go2.py -``` diff --git a/docs/old/modules_CN.md b/docs/old/modules_CN.md deleted file mode 100644 index 89e16c7112..0000000000 --- a/docs/old/modules_CN.md +++ /dev/null @@ -1,188 +0,0 @@ -# Dimensional 模块系统 - -DimOS 模块系统使用 Dask 进行计算分布和 LCM（轻量级通信和编组）进行高性能进程间通信，实现分布式、多进程的机器人应用。 - -## 核心概念 - -### 1. 模块定义 -模块是继承自 `dimos.core.Module` 的 Python 类，定义输入、输出和 RPC 方法： - -```python -from dimos.core import Module, In, Out, rpc -from dimos.msgs.geometry_msgs import Vector3 - -class MyModule(Module): # ROS Node - # 将输入/输出声明为初始化为 None 的类属性 - data_in: In[Vector3] = None # ROS Subscriber - data_out: Out[Vector3] = None # ROS Publisher - - def __init__(): - # 调用父类 Module 初始化 - super().__init__() - - @rpc - def remote_method(self, param): - """使用 @rpc 装饰的方法可以远程调用""" - return param * 2 -``` - -### 2. 模块部署 -使用 `dimos.deploy()` 方法在 Dask 工作进程中部署模块： - -```python -from dimos import core - -# 启动具有 N 个工作进程的 Dask 集群 -dimos = core.start(4) - -# 部署模块时可以传递初始化参数 -# 在这种情况下，param1 和 param2 被传递到模块初始化中 -module = dimos.deploy(Module, param1="value1", param2=123) -``` - -### 3. 流连接 -模块通过使用 LCM 传输的响应式流进行通信： - -```python -# 为输出配置 LCM 传输 -module1.data_out.transport = core.LCMTransport("/topic_name", MessageType) - -# 将模块输入连接到输出 -module2.data_in.connect(module1.data_out) - -# 访问底层的 Observable 流 -stream = module1.data_out.observable() -stream.subscribe(lambda msg: print(f"接收到: {msg}")) -``` - -### 4. 模块生命周期 -```python -# 启动模块以开始处理 -module.start() # 如果定义了 @rpc start() 方法，则调用它 - -# 检查模块 I/O 配置 -print(module.io().result()) # 显示输入、输出和 RPC 方法 - -# 优雅关闭 -dimos.shutdown() -``` - -## 实际示例：机器人控制系统 - -```python -# 连接模块封装机器人硬件/仿真 -connection = dimos.deploy(ConnectionModule, ip=robot_ip) -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) -connection.video.transport = core.LCMTransport("/video", Image) - -# 感知模块处理传感器数据 -perception = dimos.deploy(PersonTrackingStream, camera_intrinsics=[...]) -perception.video.connect(connection.video) -perception.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# 开始处理 -connection.start() -perception.start() - -# 通过 RPC 启用跟踪 -perception.enable_tracking() - -# 获取最新的跟踪数据 -data = perception.get_tracking_data() -``` - -## LCM 传输配置 - -```python -# 用于简单类型（如激光雷达）的标准 LCM 传输 -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) - -# 用于复杂 Python 对象/字典的基于 pickle 的传输 -connection.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# 自动配置 LCM 系统缓冲区（在容器中必需） -from dimos.protocol import pubsub -pubsub.lcm.autoconf() -``` - -这种架构使得能够将复杂的机器人系统构建为可组合的分布式模块，这些模块通过流和 RPC 高效通信，从单机扩展到集群。 - -# Dimensional 安装指南 -## Python 安装（Ubuntu 22.04） - -```bash -sudo apt install python3-venv - -# 克隆仓库（dev 分支，无子模块） -git clone -b dev https://github.com/dimensionalOS/dimos.git -cd dimos - -# 创建并激活虚拟环境 -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# 如果尚未安装，请安装 torch 和 torchvision -# 示例 CUDA 11.7，Pytorch 2.0.1（如果需要不同的 pytorch 版本，请替换） -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -``` - -### 安装依赖 -```bash -# 仅 CPU（建议首先尝试） -pip install .[cpu,dev] - -# CUDA 安装 -pip install .[cuda,dev] - -# 复制并配置环境变量 -cp default.env .env -``` - -### 测试安装 -```bash -# 运行标准测试 -pytest -s dimos/ - -# 测试模块功能 -pytest -s -m module dimos/ - -# 测试 LCM 通信 -pytest -s -m lcm dimos/ -``` - -# Unitree Go2 快速开始 - -要快速测试模块系统，您可以直接运行 Unitree Go2 多进程示例： - -```bash -# 确保设置了所需的环境变量 -export ROBOT_IP= - -# 运行多进程 Unitree Go2 示例 -python dimos/robot/unitree_webrtc/multiprocess/unitree_go2.py -``` - -## 模块系统的高级特性 - -### 分布式计算 -DimOS 模块系统建立在 Dask 之上，提供了强大的分布式计算能力： - -- **自动负载均衡**：模块自动分布在可用的工作进程中 -- **容错性**：如果工作进程失败，模块可以在其他工作进程上重新启动 -- **可扩展性**：从单机到集群的无缝扩展 - -### 响应式编程模型 -使用 RxPY 实现的响应式流提供了： - -- **异步处理**：非阻塞的数据流处理 -- **背压处理**：自动管理快速生产者和慢速消费者 -- **操作符链**：使用 map、filter、merge 等操作符进行流转换 - -### 性能优化 -LCM 传输针对机器人应用进行了优化： - -- **零拷贝**：大型消息的高效内存使用 -- **低延迟**：微秒级的消息传递 -- **多播支持**：一对多的高效通信 diff --git a/docs/old/ros_navigation.md b/docs/old/ros_navigation.md deleted file mode 100644 index 4a74500b2f..0000000000 --- a/docs/old/ros_navigation.md +++ /dev/null @@ -1,284 +0,0 @@ -# Autonomy Stack API Documentation - -## Prerequisites - -- Ubuntu 24.04 -- [ROS 2 Jazzy Installation](https://docs.ros.org/en/jazzy/Installation.html) - -Add the following line to your `~/.bashrc` to source the ROS 2 Jazzy setup script automatically: - -``` echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc``` - -## MID360 Ethernet Configuration (skip for sim) - -### Step 1: Configure Network Interface - -1. Open Network Settings in Ubuntu -2. Find your Ethernet connection to the MID360 -3. Click the gear icon to edit settings -4. Go to IPv4 tab -5. Change Method from "Automatic (DHCP)" to "Manual" -6. Add the following settings: - - **Address**: 192.168.1.5 - - **Netmask**: 255.255.255.0 - - **Gateway**: 192.168.1.1 -7. Click "Apply" - -### Step 2: Configure MID360 IP in JSON - -1. Find your MID360 serial number (on sticker under QR code) -2. Note the last 2 digits (e.g., if serial ends in 89, use 189) -3. Edit the configuration file: - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform -nano src/utilities/livox_ros_driver2/config/MID360_config.json -``` - -4. Update line 28 with your IP (192.168.1.1xx where xx = last 2 digits): - -```json -"ip" : "192.168.1.1xx", -``` - -5. Save and exit - -### Step 3: Verify Connection - -```bash -ping 192.168.1.1xx # Replace xx with your last 2 digits -``` - -## Robot Configuration - -### Setting Robot Type - -The system supports different robot configurations. Set the `ROBOT_CONFIG_PATH` environment variable to specify which robot configuration to use: - -```bash -# For Unitree G1 (default if not set) -export ROBOT_CONFIG_PATH="unitree/unitree_g1" - -# Add to ~/.bashrc to make permanent -echo 'export ROBOT_CONFIG_PATH="unitree/unitree_g1"' >> ~/.bashrc -``` - -Available robot configurations: -- `unitree/unitree_g1` - Unitree G1 robot (default) -- Add your custom robot configs in `src/base_autonomy/local_planner/config/` - -## Build the system - -You must do this every you make a code change, this is not Python - -```colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release``` - -## System Launch - -### Simulation Mode - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform - -# Base autonomy only -./system_simulation.sh - -# With route planner -./system_simulation_with_route_planner.sh - -# With exploration planner -./system_simulation_with_exploration_planner.sh -``` - -### Real Robot Mode - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform - -# Base autonomy only -./system_real_robot.sh - -# With route planner -./system_real_robot_with_route_planner.sh - -# With exploration planner -./system_real_robot_with_exploration_planner.sh -``` - -## Quick Troubleshooting - -- **Cannot ping MID360**: Check Ethernet cable and network settings -- **SLAM drift**: Press clear-terrain-map button on joystick controller -- **Joystick not recognized**: Unplug and replug USB dongle - - -## ROS Topics - -### Input Topics (Commands) - -| Topic | Type | Description | -|-------|------|-------------| -| `/way_point` | `geometry_msgs/PointStamped` | Send navigation goal (position only) | -| `/goal_pose` | `geometry_msgs/PoseStamped` | Send goal with orientation | -| `/cancel_goal` | `std_msgs/Bool` | Cancel current goal (data: true) | -| `/joy` | `sensor_msgs/Joy` | Joystick input | -| `/stop` | `std_msgs/Int8` | Soft Stop (2=stop all commmand, 0 = release) | -| `/navigation_boundary` | `geometry_msgs/PolygonStamped` | Set navigation boundaries | -| `/added_obstacles` | `sensor_msgs/PointCloud2` | Virtual obstacles | - -### Output Topics (Status) - -| Topic | Type | Description | -|-------|------|-------------| -| `/state_estimation` | `nav_msgs/Odometry` | Robot pose from SLAM | -| `/registered_scan` | `sensor_msgs/PointCloud2` | Aligned lidar point cloud | -| `/terrain_map` | `sensor_msgs/PointCloud2` | Local terrain map | -| `/terrain_map_ext` | `sensor_msgs/PointCloud2` | Extended terrain map | -| `/path` | `nav_msgs/Path` | Local path being followed | -| `/cmd_vel` | `geometry_msgs/Twist` | Velocity commands to motors | -| `/goal_reached` | `std_msgs/Bool` | True when goal reached, false when cancelled/new goal | - -### Map Topics - -| Topic | Type | Description | -|-------|------|-------------| -| `/overall_map` | `sensor_msgs/PointCloud2` | Global map (only in sim)| -| `/registered_scan` | `sensor_msgs/PointCloud2` | Current scan in map frame | -| `/terrain_map` | `sensor_msgs/PointCloud2` | Local obstacle map | - -## Usage Examples - -### Send Goal -```bash -ros2 topic pub /way_point geometry_msgs/msg/PointStamped "{ - header: {frame_id: 'map'}, - point: {x: 5.0, y: 3.0, z: 0.0} -}" --once -``` - -### Cancel Goal -```bash -ros2 topic pub /cancel_goal std_msgs/msg/Bool "data: true" --once -``` - -### Monitor Robot State -```bash -ros2 topic echo /state_estimation -``` - -## Configuration Parameters - -### Vehicle Parameters (`localPlanner`) - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `vehicleLength` | 0.5 | Robot length (m) | -| `vehicleWidth` | 0.5 | Robot width (m) | -| `maxSpeed` | 0.875 | Maximum speed (m/s) | -| `autonomySpeed` | 0.875 | Autonomous mode speed (m/s) | - -### Goal Tolerance Parameters - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `goalReachedThreshold` | 0.3-0.5 | Distance to consider goal reached (m) | -| `goalClearRange` | 0.35-0.6 | Extra clearance around goal (m) | -| `goalBehindRange` | 0.35-0.8 | Stop pursuing if goal behind within this distance (m) | -| `omniDirGoalThre` | 1.0 | Distance for omnidirectional approach (m) | - -### Obstacle Avoidance - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `obstacleHeightThre` | 0.1-0.2 | Height threshold for obstacles (m) | -| `adjacentRange` | 3.5 | Sensor range for planning (m) | -| `minRelZ` | -0.4 | Minimum relative height to consider (m) | -| `maxRelZ` | 0.3 | Maximum relative height to consider (m) | - -### Path Planning - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `pathScale` | 0.875 | Path resolution scale | -| `minPathScale` | 0.675 | Minimum path scale when blocked | -| `minPathRange` | 0.8 | Minimum planning range (m) | -| `dirThre` | 90.0 | Direction threshold (degrees) | - -### Control Parameters (`pathFollower`) - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `lookAheadDis` | 0.5 | Look-ahead distance (m) | -| `maxAccel` | 2.0 | Maximum acceleration (m/s²) | -| `slowDwnDisThre` | 0.875 | Slow down distance threshold (m) | - -### SLAM Blind Zones (`feature_extraction_node`) - -| Parameter | Mecanum | Description | -|-----------|---------|-------------| -| `blindFront` | 0.1 | Front blind zone (m) | -| `blindBack` | -0.2 | Back blind zone (m) | -| `blindLeft` | 0.1 | Left blind zone (m) | -| `blindRight` | -0.1 | Right blind zone (m) | -| `blindDiskRadius` | 0.4 | Cylindrical blind zone radius (m) | - -## Operating Modes - -### Mode Control -- **Joystick L2**: Hold for autonomy mode -- **Joystick R2**: Hold to disable obstacle checking - -### Speed Control -The robot automatically adjusts speed based on: -1. Obstacle proximity -2. Path complexity -3. Goal distance - -## Tuning Guide - -### For Tighter Navigation -- Decrease `goalReachedThreshold` (e.g., 0.2) -- Decrease `goalClearRange` (e.g., 0.3) -- Decrease `vehicleLength/Width` slightly - -### For Smoother Navigation -- Increase `goalReachedThreshold` (e.g., 0.5) -- Increase `lookAheadDis` (e.g., 0.7) -- Decrease `maxAccel` (e.g., 1.5) - -### For Aggressive Obstacle Avoidance -- Increase `obstacleHeightThre` (e.g., 0.15) -- Increase `adjacentRange` (e.g., 4.0) -- Increase blind zone parameters - -## Common Issues - -### Robot Oscillates at Goal -- Increase `goalReachedThreshold` -- Increase `goalBehindRange` - -### Robot Stops Too Far from Goal -- Decrease `goalReachedThreshold` -- Decrease `goalClearRange` - -### Robot Hits Low Obstacles -- Decrease `obstacleHeightThre` -- Adjust `minRelZ` to include lower points - -## SLAM Configuration - -### Localization Mode -Set in `livox_mid360.yaml`: -```yaml -local_mode: true -init_x: 0.0 -init_y: 0.0 -init_yaw: 0.0 -``` - -### Mapping Performance -```yaml -mapping_line_resolution: 0.1 # Decrease for higher quality -mapping_plane_resolution: 0.2 # Decrease for higher quality -max_iterations: 5 # Increase for better accuracy -``` diff --git a/docs/old/testing_stream_reply.md b/docs/old/testing_stream_reply.md deleted file mode 100644 index e3189bb5e8..0000000000 --- a/docs/old/testing_stream_reply.md +++ /dev/null @@ -1,174 +0,0 @@ -# Sensor Replay & Storage Toolkit - -A lightweight framework for **recording, storing, and replaying binary data streams for automated tests**. It keeps your repository small (data lives in Git LFS) while giving you Python‑first ergonomics for working with RxPY streams, point‑clouds, videos, command logs—anything you can pickle. - ---- - -## 1 At a Glance - -| Need | One liner | -| ------------------------------ | ------------------------------------------------------------- | -| **Iterate over every message** | `SensorReplay("raw_odometry_rotate_walk").iterate(print)` | -| **RxPY stream for piping** | `SensorReplay("raw_odometry_rotate_walk").stream().pipe(...)` | -| **Throttle replay rate** | `SensorReplay("raw_odometry_rotate_walk").stream(rate_hz=10)` | -| **Raw path to a blob/dir** | `path = testData("raw_odometry_rotate_walk")` | -| **Store a new stream** | see [`SensorStorage`](#5-storing-new-streams) | - -> If the requested blob is missing locally, it is transparently downloaded from Git LFS, extracted to `tests/data//`, and cached for subsequent runs. - ---- - -## 2 Goals - -* **Zero setup for CI & collaborators** – data is fetched on demand. -* **No repo bloat** – binaries live in Git LFS; the working tree stays trim. -* **Symmetric API** – `SensorReplay` ↔︎ `SensorStorage`; same name, different direction. -* **Format agnostic** – replay *anything* you can pickle (protobuf, numpy, JPEG, …). -* **Data type agnostic** – with testData("raw_odometry_rotate_walk") you get a Path object back, can be a raw video file, whole codebase, ML model etc - - ---- - -## 3 Replaying Data - -### 3.1 Iterating Messages - -```python -from sensor_tools import SensorReplay - -# Print every stored Odometry message -SensorReplay(name="raw_odometry_rotate_walk").iterate(print) -``` - -### 3.2 RxPY Streaming - -```python -from rx import operators as ops -from operator import sub, add -from dimos.utils.testing import SensorReplay, SensorStorage -from dimos.robot.unitree_webrtc.type.odometry import Odometry - -# Compute total yaw rotation (radians) - -total_rad = ( - SensorReplay("raw_odometry_rotate_walk", autocast=Odometry.from_msg) - .stream() - .pipe( - ops.map(lambda odom: odom.rot.z), - ops.pairwise(), # [1,2,3,4] -> [[1,2], [2,3], [3,4]] - ops.starmap(sub), # [sub(1,2), sub(2,3), sub(3,4)] - ops.reduce(add), - ) - .run() -) - -assert total_rad == pytest.approx(4.05, abs=0.01) -``` - -### 3.3 Lidar Mapping Example (200MB blob) - -```python -from dimos.utils.testing import SensorReplay, SensorStorage -from dimos.robot.unitree_webrtc.type.map import Map - -lidar_stream = SensorReplay("office_lidar", autocast=LidarMessage.from_msg) -map_ = Map(voxel_size=0.5) - -# Blocks until the stream is consumed -map_.consume(lidar_stream.stream()).run() - -assert map_.costmap.grid.shape == (404, 276) -``` - ---- - -## 4 Low Level Access - -If you want complete control, call **`testData(name)`** to get a `Path` to the extracted file or directory — no pickling assumptions: - -```python -absolute_path: Path = testData("some_name") -``` - -Do whatever you like: open a video file, load a model checkpoint, etc. - ---- - -## 5 Storing New Streams - -1. **Write a test marked `@pytest.mark.tool`** so CI skips it by default. -2. Use `SensorStorage` to persist the stream into `tests/data//*.pickle`. - -```python -@pytest.mark.tool -def test_store_odometry_stream(): - load_dotenv() - - robot = UnitreeGo2(ip=os.getenv("ROBOT_IP"), mode="ai") - robot.standup() - - storage = SensorStorage("raw_odometry_rotate_walk2") - storage.save_stream(robot.raw_odom_stream()) # ← records until interrupted - - try: - while True: - time.sleep(0.1) - except KeyboardInterrupt: - robot.liedown() -``` - -### 5.1 Behind the Scenes - -* Any new file/dir under `tests/data/` is treated as a **data blob**. -* `./bin/lfs_push` compresses it into `tests/data/.lfs/.tar.gz` *and* uploads it to Git LFS. -* Only the `.lfs/` archive is committed; raw binaries remain `.gitignored`. - ---- - -## 6 Storing Arbitrary Binary Data - -Just copy to `tests/data/whatever` -* `./bin/lfs_push` compresses it into `tests/data/.lfs/.tar.gz` *and* uploads it to Git LFS. - ---- - -## 7 Developer Workflow Checklist - -1. **Drop new data** into `tests/data/`. -2. Run your new tests that use SensorReplay or testData calls, make sure all works -3. Run `./bin/lfs_push` (or let the pre commit hook nag you). -4. Commit the resulting `tests/data/.lfs/.tar.gz`. -5. Optional - you can delete `tests/data/your_new_stuff` and re-run the test to ensure it gets downloaded from LFS correclty -6. Push/PR - -### 7.1 Pre commit Setup (optional but recommended) - -```sh -sudo apt install pre-commit -pre-commit install # inside repo root -``` - -Now each commit checks formatting, linting, *and* whether you forgot to push new blobs: - -``` -$ echo test > tests/data/foo.txt -$ git add tests/data/foo.txt && git commit -m "demo" -LFS data ......................................................... Failed -✗ New test data detected at /tests/data: - foo.txt -Either delete or run ./bin/lfs_push -``` - ---- - -## 8 Future Work - -- A replay rate that mirrors the **original message timestamps** can be implemented downstream (e.g., an RxPY operator) -- Likely this same system should be used for production binary data delivery as well (Models etc) - ---- - -## 9 Existing Examples - -* `dimos/robot/unitree_webrtc/type/test_odometry.py` -* `dimos/robot/unitree_webrtc/type/test_map.py` From 3a8fa53a7e02b32151bd36f608dc954f88f3de53 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 23:00:00 -0800 Subject: [PATCH 017/136] remove probably outdated docs, consolidate remaining --- dimos/hardware/README.md | 29 -- .../{README.md => manipulation.md} | 0 .../sensors/camera/gstreamer/readme.md | 1 - dimos/robot/agilex/README.md | 371 -------------- dimos/robot/agilex/README_CN.md | 465 ------------------ dimos/robot/cli/README.md | 65 --- dimos/simulation/README.md | 98 ---- dimos/web/README.md | 126 ----- dimos/web/command-center-extension/README.md | 17 - .../concepts/blueprints.md | 0 10 files changed, 1172 deletions(-) delete mode 100644 dimos/hardware/README.md rename dimos/hardware/manipulators/{README.md => manipulation.md} (100%) delete mode 100644 dimos/hardware/sensors/camera/gstreamer/readme.md delete mode 100644 dimos/robot/agilex/README.md delete mode 100644 dimos/robot/agilex/README_CN.md delete mode 100644 dimos/robot/cli/README.md delete mode 100644 dimos/simulation/README.md delete mode 100644 dimos/web/README.md delete mode 100644 dimos/web/command-center-extension/README.md rename dimos/core/README_BLUEPRINTS.md => docs/concepts/blueprints.md (100%) diff --git a/dimos/hardware/README.md b/dimos/hardware/README.md deleted file mode 100644 index 2587e3595d..0000000000 --- a/dimos/hardware/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Hardware - -## Remote camera stream with timestamps - -### Required Ubuntu packages: - -```bash -sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav python3-gi python3-gi-cairo gir1.2-gstreamer-1.0 gir1.2-gst-plugins-base-1.0 v4l-utils gstreamer1.0-vaapi -``` - -### Usage - -On sender machine (with the camera): - -```bash -python3 dimos/hardware/gstreamer_sender.py --device /dev/video0 --host 0.0.0.0 --port 5000 -``` - -If it's a stereo camera and you only want to send the left side (the left camera): - -```bash -python3 dimos/hardware/gstreamer_sender.py --device /dev/video0 --host 0.0.0.0 --port 5000 --single-camera -``` - -On receiver machine: - -```bash -python3 dimos/hardware/gstreamer_camera_test_script.py --host 10.0.0.227 --port 5000 -``` diff --git a/dimos/hardware/manipulators/README.md b/dimos/hardware/manipulators/manipulation.md similarity index 100% rename from dimos/hardware/manipulators/README.md rename to dimos/hardware/manipulators/manipulation.md diff --git a/dimos/hardware/sensors/camera/gstreamer/readme.md b/dimos/hardware/sensors/camera/gstreamer/readme.md deleted file mode 100644 index 29198aea24..0000000000 --- a/dimos/hardware/sensors/camera/gstreamer/readme.md +++ /dev/null @@ -1 +0,0 @@ -This gstreamer stuff is obsoleted but could be adopted as an alternative hardware for camera module if needed diff --git a/dimos/robot/agilex/README.md b/dimos/robot/agilex/README.md deleted file mode 100644 index 8342a6045e..0000000000 --- a/dimos/robot/agilex/README.md +++ /dev/null @@ -1,371 +0,0 @@ -# DIMOS Manipulator Robot Development Guide - -This guide explains how to create robot classes, integrate agents, and use the DIMOS module system with LCM transport. - -## Table of Contents -1. [Robot Class Architecture](#robot-class-architecture) -2. [Module System & LCM Transport](#module-system--lcm-transport) -3. [Agent Integration](#agent-integration) -4. [Complete Example](#complete-example) - -## Robot Class Architecture - -### Basic Robot Class Structure - -A DIMOS robot class should follow this pattern: - -```python -from typing import Optional, List -from dimos import core -from dimos.types.robot_capabilities import RobotCapability - -class YourRobot: - """Your robot implementation.""" - - def __init__(self, robot_capabilities: Optional[List[RobotCapability]] = None): - # Core components - self.dimos = None - self.modules = {} - self.skill_library = SkillLibrary() - - # Define capabilities - self.capabilities = robot_capabilities or [ - RobotCapability.VISION, - RobotCapability.MANIPULATION, - ] - - async def start(self): - """Start the robot modules.""" - # Initialize DIMOS with worker count - self.dimos = core.start(2) # Number of workers needed - - # Deploy modules - # ... (see Module System section) - - def stop(self): - """Stop all modules and clean up.""" - # Stop modules - # Close DIMOS - if self.dimos: - self.dimos.close() -``` - -### Key Components Explained - -1. **Initialization**: Store references to modules, skills, and capabilities -2. **Async Start**: Modules must be deployed asynchronously -3. **Proper Cleanup**: Always stop modules before closing DIMOS - -## Module System & LCM Transport - -### Understanding DIMOS Modules - -Modules are the building blocks of DIMOS robots. They: -- Process data streams (inputs) -- Produce outputs -- Can be connected together -- Communicate via LCM (Lightweight Communications and Marshalling) - -### Deploying a Module - -```python -# Deploy a camera module -self.camera = self.dimos.deploy( - ZEDModule, # Module class - camera_id=0, # Module parameters - resolution="HD720", - depth_mode="NEURAL", - fps=30, - publish_rate=30.0, - frame_id="camera_frame" -) -``` - -### Setting Up LCM Transport - -LCM transport enables inter-module communication: - -```python -# Enable LCM auto-configuration -from dimos.protocol import pubsub -pubsub.lcm.autoconf() - -# Configure output transport -self.camera.color_image.transport = core.LCMTransport( - "/camera/color_image", # Topic name - Image # Message type -) -self.camera.depth_image.transport = core.LCMTransport( - "/camera/depth_image", - Image -) -``` - -### Connecting Modules - -Connect module outputs to inputs: - -```python -# Connect manipulation module to camera outputs -self.manipulation.rgb_image.connect(self.camera.color_image) -self.manipulation.depth_image.connect(self.camera.depth_image) -self.manipulation.camera_info.connect(self.camera.camera_info) -``` - -### Module Communication Pattern - -``` -┌──────────────┐ LCM ┌────────────────┐ LCM ┌──────────────┐ -│ Camera │────────▶│ Manipulation │────────▶│ Visualization│ -│ Module │ Messages│ Module │ Messages│ Output │ -└──────────────┘ └────────────────┘ └──────────────┘ - ▲ ▲ - │ │ - └──────────────────────────┘ - Direct Connection via RPC call -``` - -## Agent Integration - -### Setting Up Agent with Robot - -The run file pattern for agent integration: - -```python -#!/usr/bin/env python3 -import asyncio -import reactivex as rx -from dimos.agents_deprecated.claude_agent import ClaudeAgent -from dimos.web.robot_web_interface import RobotWebInterface - -def main(): - # 1. Create and start robot - robot = YourRobot() - asyncio.run(robot.start()) - - # 2. Set up skills - skills = robot.get_skills() - skills.add(YourSkill) - skills.create_instance("YourSkill", robot=robot) - - # 3. Set up reactive streams - agent_response_subject = rx.subject.Subject() - agent_response_stream = agent_response_subject.pipe(ops.share()) - - # 4. Create web interface - web_interface = RobotWebInterface( - port=5555, - text_streams={"agent_responses": agent_response_stream}, - audio_subject=rx.subject.Subject() - ) - - # 5. Create agent - agent = ClaudeAgent( - dev_name="your_agent", - input_query_stream=web_interface.query_stream, - skills=skills, - system_query="Your system prompt here", - model_name="claude-3-5-haiku-latest" - ) - - # 6. Connect agent responses - agent.get_response_observable().subscribe( - lambda x: agent_response_subject.on_next(x) - ) - - # 7. Run interface - web_interface.run() -``` - -### Key Integration Points - -1. **Reactive Streams**: Use RxPy for event-driven communication -2. **Web Interface**: Provides user input/output -3. **Agent**: Processes natural language and executes skills -4. **Skills**: Define robot capabilities as executable actions - -## Complete Example - -### Step 1: Create Robot Class (`my_robot.py`) - -```python -import asyncio -from typing import Optional, List -from dimos import core -from dimos.hardware.camera import CameraModule -from dimos.manipulation.module import ManipulationModule -from dimos.skills.skills import SkillLibrary -from dimos.types.robot_capabilities import RobotCapability -from dimos_lcm.sensor_msgs import Image, CameraInfo -from dimos.protocol import pubsub - -class MyRobot: - def __init__(self, robot_capabilities: Optional[List[RobotCapability]] = None): - self.dimos = None - self.camera = None - self.manipulation = None - self.skill_library = SkillLibrary() - - self.capabilities = robot_capabilities or [ - RobotCapability.VISION, - RobotCapability.MANIPULATION, - ] - - async def start(self): - # Start DIMOS - self.dimos = core.start(2) - - # Enable LCM - pubsub.lcm.autoconf() - - # Deploy camera - self.camera = self.dimos.deploy( - CameraModule, - camera_id=0, - fps=30 - ) - - # Configure camera LCM - self.camera.color_image.transport = core.LCMTransport("/camera/rgb", Image) - self.camera.depth_image.transport = core.LCMTransport("/camera/depth", Image) - self.camera.camera_info.transport = core.LCMTransport("/camera/info", CameraInfo) - - # Deploy manipulation - self.manipulation = self.dimos.deploy(ManipulationModule) - - # Connect modules - self.manipulation.rgb_image.connect(self.camera.color_image) - self.manipulation.depth_image.connect(self.camera.depth_image) - self.manipulation.camera_info.connect(self.camera.camera_info) - - # Configure manipulation output - self.manipulation.viz_image.transport = core.LCMTransport("/viz/output", Image) - - # Start modules - self.camera.start() - self.manipulation.start() - - await asyncio.sleep(2) # Allow initialization - - def get_skills(self): - return self.skill_library - - def stop(self): - if self.manipulation: - self.manipulation.stop() - if self.camera: - self.camera.stop() - if self.dimos: - self.dimos.close() -``` - -### Step 2: Create Run Script (`run.py`) - -```python -#!/usr/bin/env python3 -import asyncio -import os -from my_robot import MyRobot -from dimos.agents_deprecated.claude_agent import ClaudeAgent -from dimos.skills.basic import BasicSkill -from dimos.web.robot_web_interface import RobotWebInterface -import reactivex as rx -import reactivex.operators as ops - -SYSTEM_PROMPT = """You are a helpful robot assistant.""" - -def main(): - # Check API key - if not os.getenv("ANTHROPIC_API_KEY"): - print("Please set ANTHROPIC_API_KEY") - return - - # Create robot - robot = MyRobot() - - try: - # Start robot - asyncio.run(robot.start()) - - # Set up skills - skills = robot.get_skills() - skills.add(BasicSkill) - skills.create_instance("BasicSkill", robot=robot) - - # Set up streams - agent_response_subject = rx.subject.Subject() - agent_response_stream = agent_response_subject.pipe(ops.share()) - - # Create web interface - web_interface = RobotWebInterface( - port=5555, - text_streams={"agent_responses": agent_response_stream} - ) - - # Create agent - agent = ClaudeAgent( - dev_name="my_agent", - input_query_stream=web_interface.query_stream, - skills=skills, - system_query=SYSTEM_PROMPT - ) - - # Connect responses - agent.get_response_observable().subscribe( - lambda x: agent_response_subject.on_next(x) - ) - - print("Robot ready at http://localhost:5555") - - # Run - web_interface.run() - - finally: - robot.stop() - -if __name__ == "__main__": - main() -``` - -### Step 3: Define Skills (`skills.py`) - -```python -from dimos.skills import Skill, skill - -@skill( - description="Perform a basic action", - parameters={ - "action": "The action to perform" - } -) -class BasicSkill(Skill): - def __init__(self, robot): - self.robot = robot - - def run(self, action: str): - # Implement skill logic - return f"Performed: {action}" -``` - -## Best Practices - -1. **Module Lifecycle**: Always start DIMOS before deploying modules -2. **LCM Topics**: Use descriptive topic names with namespaces -3. **Error Handling**: Wrap module operations in try-except blocks -4. **Resource Cleanup**: Ensure proper cleanup in stop() methods -5. **Async Operations**: Use asyncio for non-blocking operations -6. **Stream Management**: Use RxPy for reactive programming patterns - -## Debugging Tips - -1. **Check Module Status**: Print module.io().result() to see connections -2. **Monitor LCM**: Use Foxglove to visualize LCM messages -3. **Log Everything**: Use dimos.utils.logging_config.setup_logger() -4. **Test Modules Independently**: Deploy and test one module at a time - -## Common Issues - -1. **"Module not started"**: Ensure start() is called after deployment -2. **"No data received"**: Check LCM transport configuration -3. **"Connection failed"**: Verify input/output types match -4. **"Cleanup errors"**: Stop modules before closing DIMOS diff --git a/dimos/robot/agilex/README_CN.md b/dimos/robot/agilex/README_CN.md deleted file mode 100644 index a8d79ebec1..0000000000 --- a/dimos/robot/agilex/README_CN.md +++ /dev/null @@ -1,465 +0,0 @@ -# DIMOS 机械臂机器人开发指南 - -本指南介绍如何创建机器人类、集成智能体（Agent）以及使用 DIMOS 模块系统和 LCM 传输。 - -## 目录 -1. [机器人类架构](#机器人类架构) -2. [模块系统与 LCM 传输](#模块系统与-lcm-传输) -3. [智能体集成](#智能体集成) -4. [完整示例](#完整示例) - -## 机器人类架构 - -### 基本机器人类结构 - -DIMOS 机器人类应遵循以下模式： - -```python -from typing import Optional, List -from dimos import core -from dimos.types.robot_capabilities import RobotCapability - -class YourRobot: - """您的机器人实现。""" - - def __init__(self, robot_capabilities: Optional[List[RobotCapability]] = None): - # 核心组件 - self.dimos = None - self.modules = {} - self.skill_library = SkillLibrary() - - # 定义能力 - self.capabilities = robot_capabilities or [ - RobotCapability.VISION, - RobotCapability.MANIPULATION, - ] - - async def start(self): - """启动机器人模块。""" - # 初始化 DIMOS，指定工作线程数 - self.dimos = core.start(2) # 需要的工作线程数 - - # 部署模块 - # ... (参见模块系统章节) - - def stop(self): - """停止所有模块并清理资源。""" - # 停止模块 - # 关闭 DIMOS - if self.dimos: - self.dimos.close() -``` - -### 关键组件说明 - -1. **初始化**：存储模块、技能和能力的引用 -2. **异步启动**：模块必须异步部署 -3. **正确清理**：在关闭 DIMOS 之前始终停止模块 - -## 模块系统与 LCM 传输 - -### 理解 DIMOS 模块 - -模块是 DIMOS 机器人的构建块。它们： -- 处理数据流（输入） -- 产生输出 -- 可以相互连接 -- 通过 LCM（轻量级通信和编组）进行通信 - -### 部署模块 - -```python -# 部署相机模块 -self.camera = self.dimos.deploy( - ZEDModule, # 模块类 - camera_id=0, # 模块参数 - resolution="HD720", - depth_mode="NEURAL", - fps=30, - publish_rate=30.0, - frame_id="camera_frame" -) -``` - -### 设置 LCM 传输 - -LCM 传输实现模块间通信： - -```python -# 启用 LCM 自动配置 -from dimos.protocol import pubsub -pubsub.lcm.autoconf() - -# 配置输出传输 -self.camera.color_image.transport = core.LCMTransport( - "/camera/color_image", # 主题名称 - Image # 消息类型 -) -self.camera.depth_image.transport = core.LCMTransport( - "/camera/depth_image", - Image -) -``` - -### 连接模块 - -将模块输出连接到输入： - -```python -# 将操作模块连接到相机输出 -self.manipulation.rgb_image.connect(self.camera.color_image) # ROS set_callback -self.manipulation.depth_image.connect(self.camera.depth_image) -self.manipulation.camera_info.connect(self.camera.camera_info) -``` - -### 模块通信模式 - -``` -┌──────────────┐ LCM ┌────────────────┐ LCM ┌──────────────┐ -│ 相机模块 │────────▶│ 操作模块 │────────▶│ 可视化输出 │ -│ │ 消息 │ │ 消息 │ │ -└──────────────┘ └────────────────┘ └──────────────┘ - ▲ ▲ - │ │ - └──────────────────────────┘ - 直接连接(RPC指令) -``` - -## 智能体集成 - -### 设置智能体与机器人 - -运行文件的智能体集成模式： - -```python -#!/usr/bin/env python3 -import asyncio -import reactivex as rx -from dimos.agents_deprecated.claude_agent import ClaudeAgent -from dimos.web.robot_web_interface import RobotWebInterface - -def main(): - # 1. 创建并启动机器人 - robot = YourRobot() - asyncio.run(robot.start()) - - # 2. 设置技能 - skills = robot.get_skills() - skills.add(YourSkill) - skills.create_instance("YourSkill", robot=robot) - - # 3. 设置响应式流 - agent_response_subject = rx.subject.Subject() - agent_response_stream = agent_response_subject.pipe(ops.share()) - - # 4. 创建 Web 界面 - web_interface = RobotWebInterface( - port=5555, - text_streams={"agent_responses": agent_response_stream}, - audio_subject=rx.subject.Subject() - ) - - # 5. 创建智能体 - agent = ClaudeAgent( - dev_name="your_agent", - input_query_stream=web_interface.query_stream, - skills=skills, - system_query="您的系统提示词", - model_name="claude-3-5-haiku-latest" - ) - - # 6. 连接智能体响应 - agent.get_response_observable().subscribe( - lambda x: agent_response_subject.on_next(x) - ) - - # 7. 运行界面 - web_interface.run() -``` - -### 关键集成点 - -1. **响应式流**：使用 RxPy 进行事件驱动通信 -2. **Web 界面**：提供用户输入/输出 -3. **智能体**：处理自然语言并执行技能 -4. **技能**：将机器人能力定义为可执行动作 - -## 完整示例 - -### 步骤 1：创建机器人类（`my_robot.py`） - -```python -import asyncio -from typing import Optional, List -from dimos import core -from dimos.hardware.camera import CameraModule -from dimos.manipulation.module import ManipulationModule -from dimos.skills.skills import SkillLibrary -from dimos.types.robot_capabilities import RobotCapability -from dimos_lcm.sensor_msgs import Image, CameraInfo -from dimos.protocol import pubsub - -class MyRobot: - def __init__(self, robot_capabilities: Optional[List[RobotCapability]] = None): - self.dimos = None - self.camera = None - self.manipulation = None - self.skill_library = SkillLibrary() - - self.capabilities = robot_capabilities or [ - RobotCapability.VISION, - RobotCapability.MANIPULATION, - ] - - async def start(self): - # 启动 DIMOS - self.dimos = core.start(2) - - # 启用 LCM - pubsub.lcm.autoconf() - - # 部署相机 - self.camera = self.dimos.deploy( - CameraModule, - camera_id=0, - fps=30 - ) - - # 配置相机 LCM - self.camera.color_image.transport = core.LCMTransport("/camera/rgb", Image) - self.camera.depth_image.transport = core.LCMTransport("/camera/depth", Image) - self.camera.camera_info.transport = core.LCMTransport("/camera/info", CameraInfo) - - # 部署操作模块 - self.manipulation = self.dimos.deploy(ManipulationModule) - - # 连接模块 - self.manipulation.rgb_image.connect(self.camera.color_image) - self.manipulation.depth_image.connect(self.camera.depth_image) - self.manipulation.camera_info.connect(self.camera.camera_info) - - # 配置操作输出 - self.manipulation.viz_image.transport = core.LCMTransport("/viz/output", Image) - - # 启动模块 - self.camera.start() - self.manipulation.start() - - await asyncio.sleep(2) # 允许初始化 - - def get_skills(self): - return self.skill_library - - def stop(self): - if self.manipulation: - self.manipulation.stop() - if self.camera: - self.camera.stop() - if self.dimos: - self.dimos.close() -``` - -### 步骤 2：创建运行脚本（`run.py`） - -```python -#!/usr/bin/env python3 -import asyncio -import os -from my_robot import MyRobot -from dimos.agents_deprecated.claude_agent import ClaudeAgent -from dimos.skills.basic import BasicSkill -from dimos.web.robot_web_interface import RobotWebInterface -import reactivex as rx -import reactivex.operators as ops - -SYSTEM_PROMPT = """您是一个有用的机器人助手。""" - -def main(): - # 检查 API 密钥 - if not os.getenv("ANTHROPIC_API_KEY"): - print("请设置 ANTHROPIC_API_KEY") - return - - # 创建机器人 - robot = MyRobot() - - try: - # 启动机器人 - asyncio.run(robot.start()) - - # 设置技能 - skills = robot.get_skills() - skills.add(BasicSkill) - skills.create_instance("BasicSkill", robot=robot) - - # 设置流 - agent_response_subject = rx.subject.Subject() - agent_response_stream = agent_response_subject.pipe(ops.share()) - - # 创建 Web 界面 - web_interface = RobotWebInterface( - port=5555, - text_streams={"agent_responses": agent_response_stream} - ) - - # 创建智能体 - agent = ClaudeAgent( - dev_name="my_agent", - input_query_stream=web_interface.query_stream, - skills=skills, - system_query=SYSTEM_PROMPT - ) - - # 连接响应 - agent.get_response_observable().subscribe( - lambda x: agent_response_subject.on_next(x) - ) - - print("机器人就绪，访问 http://localhost:5555") - - # 运行 - web_interface.run() - - finally: - robot.stop() - -if __name__ == "__main__": - main() -``` - -### 步骤 3：定义技能（`skills.py`） - -```python -from dimos.skills import Skill, skill - -@skill( - description="执行一个基本动作", - parameters={ - "action": "要执行的动作" - } -) -class BasicSkill(Skill): - def __init__(self, robot): - self.robot = robot - - def run(self, action: str): - # 实现技能逻辑 - return f"已执行：{action}" -``` - -## 最佳实践 - -1. **模块生命周期**：在部署模块之前始终先启动 DIMOS -2. **LCM 主题**：使用带命名空间的描述性主题名称 -3. **错误处理**：用 try-except 块包装模块操作 -4. **资源清理**：确保在 stop() 方法中正确清理 -5. **异步操作**：使用 asyncio 进行非阻塞操作 -6. **流管理**：使用 RxPy 进行响应式编程模式 - -## 调试技巧 - -1. **检查模块状态**：打印 module.io().result() 查看连接 -2. **监控 LCM**：使用 Foxglove 可视化 LCM 消息 -3. **记录一切**：使用 dimos.utils.logging_config.setup_logger() -4. **独立测试模块**：一次部署和测试一个模块 - -## 常见问题 - -1. **"模块未启动"**：确保在部署后调用 start() -2. **"未收到数据"**：检查 LCM 传输配置 -3. **"连接失败"**：验证输入/输出类型是否匹配 -4. **"清理错误"**：在关闭 DIMOS 之前停止模块 - -## 高级主题 - -### 自定义模块开发 - -创建自定义模块的基本结构： - -```python -from dimos.core import Module, In, Out, rpc - -class CustomModule(Module): - # 定义输入 - input_data: In[DataType] - - # 定义输出 - output_data: Out[DataType] - - def __init__(self, param1, param2, **kwargs): - super().__init__(**kwargs) - self.param1 = param1 - self.param2 = param2 - - @rpc - def start(self): - """启动模块处理。""" - self.input_data.subscribe(self._process_data) - - def _process_data(self, data): - """处理输入数据。""" - # 处理逻辑 - result = self.process(data) - # 发布输出 - self.output_data.publish(result) - - @rpc - def stop(self): - """停止模块。""" - # 清理资源 - pass -``` - -### 技能开发指南 - -技能是机器人可执行的高级动作： - -```python -from dimos.skills import Skill, skill -from typing import Optional - -@skill( - description="复杂操作技能", - parameters={ - "target": "目标对象", - "location": "目标位置" - } -) -class ComplexSkill(Skill): - def __init__(self, robot, **kwargs): - super().__init__(**kwargs) - self.robot = robot - - def run(self, target: str, location: Optional[str] = None): - """执行技能逻辑。""" - try: - # 1. 感知阶段 - object_info = self.robot.detect_object(target) - - # 2. 规划阶段 - if location: - plan = self.robot.plan_movement(object_info, location) - - # 3. 执行阶段 - result = self.robot.execute_plan(plan) - - return { - "success": True, - "message": f"成功移动 {target} 到 {location}" - } - - except Exception as e: - return { - "success": False, - "error": str(e) - } -``` - -### 性能优化 - -1. **并行处理**：使用多个工作线程处理不同模块 -2. **数据缓冲**：为高频数据流实现缓冲机制 -3. **延迟加载**：仅在需要时初始化重型模块 -4. **资源池化**：重用昂贵的资源（如神经网络模型） - -希望本指南能帮助您快速上手 DIMOS 机器人开发！ diff --git a/dimos/robot/cli/README.md b/dimos/robot/cli/README.md deleted file mode 100644 index 63087f48b8..0000000000 --- a/dimos/robot/cli/README.md +++ /dev/null @@ -1,65 +0,0 @@ -# Robot CLI - -To avoid having so many runfiles, I created a common script to run any blueprint. - -For example, to run the standard Unitree Go2 blueprint run: - -```bash -dimos run unitree-go2 -``` - -For the one with agents run: - -```bash -dimos run unitree-go2-agentic -``` - -You can dynamically connect additional modules. For example: - -```bash -dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill -``` - -## Definitions - -Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: - -```python -all_blueprints = { - "unitree-go2": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:standard", - "unitree-go2-agentic": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:agentic", - ... -} -``` - -(They are defined as imports to avoid triggering unrelated imports.) - -## `GlobalConfig` - -This tool also initializes the global config and passes it to the blueprint. - -`GlobalConfig` contains configuration options that are useful across many modules. For example: - -```python -class GlobalConfig(BaseSettings): - robot_ip: str | None = None - simulation: bool = False - replay: bool = False - n_dask_workers: int = 2 -``` - -Configuration values can be set from multiple places in order of precedence (later entries override earlier ones): - -- Default value defined on GlobalConfig. (`simulation = False`) -- Value defined in `.env` (`SIMULATION=true`) -- Value in the environment variable (`SIMULATION=true`) -- Value defined on the blueprint (`blueprint.global_config(simulation=True)`) -- Value coming from the CLI (`--simulation` or `--no-simulation`) - -For environment variables/`.env` values, you have to prefix the name with `DIMOS_`. - -For the command line, you call it like this: - -```bash -dimos --simulation run unitree-go2 -``` diff --git a/dimos/simulation/README.md b/dimos/simulation/README.md deleted file mode 100644 index 95d8b4cda1..0000000000 --- a/dimos/simulation/README.md +++ /dev/null @@ -1,98 +0,0 @@ -# Dimensional Streaming Setup - -This guide explains how to set up and run the Isaac Sim and Genesis streaming functionality via Docker. The setup is tested on Ubuntu 22.04 (recommended). - -## Prerequisites - -1. **NVIDIA Driver** - - NVIDIA Driver 535 must be installed - - Check your driver: `nvidia-smi` - - If not installed: - ```bash - sudo apt-get update - sudo apt install build-essential -y - sudo apt-get install -y nvidia-driver-535 - sudo reboot - ``` - -2. **CUDA Toolkit** - ```bash - sudo apt install -y nvidia-cuda-toolkit - ``` - -3. **Docker** - ```bash - # Install Docker - curl -fsSL https://get.docker.com -o get-docker.sh - sudo sh get-docker.sh - - # Post-install steps - sudo groupadd docker - sudo usermod -aG docker $USER - newgrp docker - ``` - -4. **NVIDIA Container Toolkit** - ```bash - # Add NVIDIA Container Toolkit repository - curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg - curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ - sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ - sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list - sudo apt-get update - - # Install the toolkit - sudo apt-get install -y nvidia-container-toolkit - sudo systemctl restart docker - - # Configure runtime - sudo nvidia-ctk runtime configure --runtime=docker - sudo systemctl restart docker - - # Verify installation - sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi - ``` - -5. **Pull Isaac Sim Image** - ```bash - sudo docker pull nvcr.io/nvidia/isaac-sim:4.2.0 - ``` - -6. **TO DO: Add ROS2 websocket server for client-side streaming** - -## Running the Streaming Example - -1. **Navigate to the docker/simulation directory** - ```bash - cd docker/simulation - ``` - -2. **Build and run with docker-compose** - For Isaac Sim: - ```bash - docker compose -f isaac/docker-compose.yml build - docker compose -f isaac/docker-compose.yml up - - ``` - - For Genesis: - ```bash - docker compose -f genesis/docker-compose.yml build - docker compose -f genesis/docker-compose.yml up - - ``` - -This will: -- Build the dimos_simulator image with ROS2 and required dependencies -- Start the MediaMTX RTSP server -- Run the test streaming example from either: - - `/tests/isaacsim/stream_camera.py` for Isaac Sim - - `/tests/genesissim/stream_camera.py` for Genesis - -## Viewing the Stream - -The camera stream will be available at: - -- RTSP: `rtsp://localhost:8554/stream` or `rtsp://:8554/stream` - -You can view it using VLC or any RTSP-capable player. diff --git a/dimos/web/README.md b/dimos/web/README.md deleted file mode 100644 index 28f418bb55..0000000000 --- a/dimos/web/README.md +++ /dev/null @@ -1,126 +0,0 @@ -# DimOS Robot Web Interface - -A streamlined interface for controlling and interacting with robots through DimOS. - -## Setup - -First, create an `.env` file in the root dimos directory with your configuration: - -```bash -# Example .env file -OPENAI_API_KEY=sk-your-openai-api-key -ROBOT_IP=192.168.x.x -CONN_TYPE=webrtc -WEBRTC_SERVER_HOST=0.0.0.0 -WEBRTC_SERVER_PORT=9991 -DISPLAY=:0 -``` - -## Unitree Go2 Example - -Running a full stack for Unitree Go2 requires three components: - -### 1. Start ROS2 Robot Driver - -```bash -# Source ROS environment -source /opt/ros/humble/setup.bash -source ~/your_ros_workspace/install/setup.bash - -# Launch robot driver -ros2 launch go2_robot_sdk robot.launch.py -``` - -### 2. Start DimOS Backend - -```bash -# In a new terminal, source your Python environment -source venv/bin/activate # Or your environment - -# Install requirements -pip install -r requirements.txt - -# Source ROS workspace (needed for robot communication) -source /opt/ros/humble/setup.bash -source ~/your_ros_workspace/install/setup.bash - -# Run the server with Robot() and Agent() initialization -python tests/test_unitree_agent_queries_fastapi.py -``` - -### 3. Start Frontend - -**Install yarn if not already installed** - -```bash -npm install -g yarn -``` - -**Then install dependencies and start the development server** - -```bash -# In a new terminal -cd dimos/web/dimos-interface - -# Install dependencies (first time only) -yarn install - -# Start development server -yarn dev -``` - -The frontend will be available at http://localhost:3000 - -## Using the Interface - -1. Access the web terminal at http://localhost:3000 -2. Type commands to control your robot: - - `unitree command ` - Send a command to the robot - - `unitree status` - Check connection status - - `unitree start_stream` - Start the video stream - - `unitree stop_stream` - Stop the video stream - -## Integrating DimOS with the DimOS-interface - -### Unitree Go2 Example - -```python -from dimos.agents_deprecated.agent import OpenAIAgent -from dimos.robot.unitree.unitree_go2 import UnitreeGo2 -from dimos.robot.unitree.unitree_skills import MyUnitreeSkills -from dimos.web.robot_web_interface import RobotWebInterface - -robot_ip = os.getenv("ROBOT_IP") - -# Initialize robot -logger.info("Initializing Unitree Robot") -robot = UnitreeGo2(ip=robot_ip, - connection_method=connection_method, - output_dir=output_dir) - -# Set up video stream -logger.info("Starting video stream") -video_stream = robot.get_ros_video_stream() - -# Create FastAPI server with video stream -logger.info("Initializing FastAPI server") -streams = {"unitree_video": video_stream} -web_interface = RobotWebInterface(port=5555, **streams) - -# Initialize agent with robot skills -skills_instance = MyUnitreeSkills(robot=robot) - -agent = OpenAIAgent( - dev_name="UnitreeQueryPerceptionAgent", - input_query_stream=web_interface.query_stream, - output_dir=output_dir, - skills=skills_instance, -) - -web_interface.run() -``` - -## Architecture - -- **Backend**: FastAPI server runs on port 5555 -- **Frontend**: Web application runs on port 3000 diff --git a/dimos/web/command-center-extension/README.md b/dimos/web/command-center-extension/README.md deleted file mode 100644 index efee4ec11d..0000000000 --- a/dimos/web/command-center-extension/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# command-center-extension - -This is a Foxglove extension for visualizing robot data and controlling the robot. See `dimos/web/websocket_vis/README.md` for how to use the module in your robot. - -## Build and use - -Install the Foxglove Studio desktop application. - -Install the Node dependencies: - - npm install - -Build the package and install it into Foxglove: - - npm run build && npm run local-install - -To add the panel, go to Foxglove Studio, click on the "Add panel" icon on the top right and select "command-center [local]". diff --git a/dimos/core/README_BLUEPRINTS.md b/docs/concepts/blueprints.md similarity index 100% rename from dimos/core/README_BLUEPRINTS.md rename to docs/concepts/blueprints.md From a6f4fc4fe0d40cdcbd8e5edfa110f2204e3993fb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 23:00:32 -0800 Subject: [PATCH 018/136] standardize naming --- docs/development/dimos_run.md | 65 +++++++++++++++++++ docs/{VIEWER_BACKENDS.md => visualization.md} | 0 2 files changed, 65 insertions(+) create mode 100644 docs/development/dimos_run.md rename docs/{VIEWER_BACKENDS.md => visualization.md} (100%) diff --git a/docs/development/dimos_run.md b/docs/development/dimos_run.md new file mode 100644 index 0000000000..63087f48b8 --- /dev/null +++ b/docs/development/dimos_run.md @@ -0,0 +1,65 @@ +# Robot CLI + +To avoid having so many runfiles, I created a common script to run any blueprint. + +For example, to run the standard Unitree Go2 blueprint run: + +```bash +dimos run unitree-go2 +``` + +For the one with agents run: + +```bash +dimos run unitree-go2-agentic +``` + +You can dynamically connect additional modules. For example: + +```bash +dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill +``` + +## Definitions + +Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: + +```python +all_blueprints = { + "unitree-go2": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:standard", + "unitree-go2-agentic": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:agentic", + ... +} +``` + +(They are defined as imports to avoid triggering unrelated imports.) + +## `GlobalConfig` + +This tool also initializes the global config and passes it to the blueprint. + +`GlobalConfig` contains configuration options that are useful across many modules. For example: + +```python +class GlobalConfig(BaseSettings): + robot_ip: str | None = None + simulation: bool = False + replay: bool = False + n_dask_workers: int = 2 +``` + +Configuration values can be set from multiple places in order of precedence (later entries override earlier ones): + +- Default value defined on GlobalConfig. (`simulation = False`) +- Value defined in `.env` (`SIMULATION=true`) +- Value in the environment variable (`SIMULATION=true`) +- Value defined on the blueprint (`blueprint.global_config(simulation=True)`) +- Value coming from the CLI (`--simulation` or `--no-simulation`) + +For environment variables/`.env` values, you have to prefix the name with `DIMOS_`. + +For the command line, you call it like this: + +```bash +dimos --simulation run unitree-go2 +``` diff --git a/docs/VIEWER_BACKENDS.md b/docs/visualization.md similarity index 100% rename from docs/VIEWER_BACKENDS.md rename to docs/visualization.md From 5e9ce4edc860e8c8817a86f1ec9d0f3f41028ac4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 23:00:53 -0800 Subject: [PATCH 019/136] typo and linking --- docs/development/main.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/development/main.md b/docs/development/main.md index 050982cc57..e781eed6d8 100644 --- a/docs/development/main.md +++ b/docs/development/main.md @@ -1,7 +1,7 @@ # Development Environment Guide 1. Pick your setup (system install, dev container, docker, nix) -2. Best How to test/hack dimos (overview and our practices) +2. How to test/hack dimos (overview and our practices) 3. How to make a PR @@ -217,8 +217,10 @@ uv run pytest dimos # 2. How to Test and Modify DimOS -## Where is `` located? +## Where is `` located? (Architecture) +* If you want to add a `dimos run ` command see [dimos_run.md](docs/development/dimos_run.md) +* For edits to manipulation see [manipulation.md](docs/development/manipulation.md) and [manipulation base](dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. @@ -251,12 +253,12 @@ You can enable a tag by selecting -m - these are configured in `./pyp # 3. How to Make a PR - Open the PR against the `dev` branch (not `main`) - **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) -- If you're writing documentation, see [writing_docs.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/writing_docs.md) for how to write code blocks. +- If you're writing documentation, see [writing_docs.md](docs/development/writing_docs.md) for how to write code blocks. - Less changed files = better - If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large_files.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/large_files.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large_files.md](docs/development/large_files.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Smaller PR's are better. From a6aca30a6b3a849cec2b6272aade9e993bdc409b Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Wed, 14 Jan 2026 23:01:02 -0800 Subject: [PATCH 020/136] linking fix --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 620d76469a..46ba6133c8 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,7 @@ if __name__ == "__main__": ).build().loop() ``` -### Note: Many More Examples in the [Examples Folder](https://github.com/dimensionalOS/dimos/tree/main/examples) +### Note: Many More Examples in the [Examples Folder](./examples) # How does DimOS work conceptually? @@ -183,7 +183,7 @@ There are several tools: For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. -If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](https://github.com/dimensionalOS/dimos/tree/main/docs/development/main.md). +If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](docs/development/main.md). ## Quickstart Development (For Hacking, not Proper Development) From 0c6352de3f81ad3f0d99bc26da93299e1ef0db2c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 00:28:58 -0800 Subject: [PATCH 021/136] cleanup todo's --- README.md | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/README.md b/README.md index 46ba6133c8..c168a87801 100644 --- a/README.md +++ b/README.md @@ -64,8 +64,6 @@ export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE dimos run unitree-go2 ``` - - #### Get it working in an interactive simulation! ```bash @@ -74,9 +72,6 @@ export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors dimos --simulation run unitree-go2 # open http://localhost:7779/command-center in your browser to control the robot movement ``` - - - #### Have it controlled by AI! @@ -89,17 +84,6 @@ dimos run unitree-go2-agentic # ex: tell it to explore the room, then tell it to go to where it can see a door humancli ``` - - - - - - - - - # How do I use it as a library? @@ -172,7 +156,7 @@ if __name__ == "__main__": # How does DimOS work conceptually? There are several tools: -- Modules: parallel processing, several modules run. Modules are usually python classes but its possible to use C++, Rust, Typescript, etc to write a module. +- [Modules](): The building blocks of DimOS, modules run in parallel and are defined in python as classes. Note: it is possible to use C++, Rust, Typescript, etc to write a module. - Streams: How modules communicate, a Pub / Sub system. - Blueprints: a way to group modules together and define their connections to each other - RPC: how one module can call a method on another module (arguments get serialized to JSON-like binary data) From 1296bc70d09a0fb93405cff9780ab18dc5d48dff Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:03:27 -0800 Subject: [PATCH 022/136] move viz docs --- docs/{ => api}/visualization.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{ => api}/visualization.md (100%) diff --git a/docs/visualization.md b/docs/api/visualization.md similarity index 100% rename from docs/visualization.md rename to docs/api/visualization.md From 55c88719b779a36728c27e65a8ea621749d11925 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:06:40 -0800 Subject: [PATCH 023/136] add note to dimos run --- docs/development/dimos_run.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/docs/development/dimos_run.md b/docs/development/dimos_run.md index 63087f48b8..9fb0b5845e 100644 --- a/docs/development/dimos_run.md +++ b/docs/development/dimos_run.md @@ -1,6 +1,20 @@ -# Robot CLI +# DimOS Run -To avoid having so many runfiles, I created a common script to run any blueprint. +#### Warning: If you just want to run a blueprint you don't need to add it to `dimos run`: + +`your_code.py` +```py +from dimos.robot.unitree_webrtc.unitree_go2_blueprints import basic as example_blueprint + +if __name__ == "__main__": + example_blueprint.build().loop() +``` + +```sh +python ./your_code.py +``` + +## Usage For example, to run the standard Unitree Go2 blueprint run: @@ -20,7 +34,7 @@ You can dynamically connect additional modules. For example: dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill ``` -## Definitions +## Adding your own Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: From b04252140a314f8957b461fff2b9eeed001ba470 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:07:53 -0800 Subject: [PATCH 024/136] alignment --- docs/development/writing_docs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/writing_docs.md b/docs/development/writing_docs.md index 6a73d0acf8..b9ae0b27df 100644 --- a/docs/development/writing_docs.md +++ b/docs/development/writing_docs.md @@ -225,8 +225,8 @@ md-babel-py run document.md --dry-run ## Running ```sh skip -md-babel-py run document.md # edit in-place -md-babel-py run document.md --stdout # preview to stdout +md-babel-py run document.md # edit in-place +md-babel-py run document.md --stdout # preview to stdout md-babel-py run document.md --dry-run # show what would run ``` From 18e763c97f51dc14c1c82432c4e9ac0e81adf6d7 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:15:00 -0800 Subject: [PATCH 025/136] rename main dev docs --- docs/development/{main.md => README.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/development/{main.md => README.md} (100%) diff --git a/docs/development/main.md b/docs/development/README.md similarity index 100% rename from docs/development/main.md rename to docs/development/README.md From 22dc1da2e4d6a51f0ee04a2bed9c1c395df26c2b Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:17:53 -0800 Subject: [PATCH 026/136] disable dev container until fixed --- docs/development/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index e781eed6d8..4743d7f46a 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -73,7 +73,8 @@ uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git uv add git+https://github.com/facebookresearch/detectron2.git ``` -## Setup Option B: Dev Containers (Recommended) + + -## Setup Option C: Nix Flake + direnv +## Setup Option B: Nix Flake + direnv ### Why pick this setup? (pros/cons/when-to-use) @@ -171,7 +172,7 @@ uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' uv run pytest dimos ``` -## Setup Option D: Nix Flake - Isolated/Reliable +## Setup Option C: Nix Flake - Isolated/Reliable ### Why pick this setup? (pros/cons/when-to-use) From e05cef184a5a2d222623b1c1325853171ade16a3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:32:06 -0800 Subject: [PATCH 027/136] - --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c168a87801..04c31aaa5c 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,7 @@ There are several tools: For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. -If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](docs/development/main.md). +If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](docs/development/README.md). ## Quickstart Development (For Hacking, not Proper Development) From 2c940d5b20d81a12401785488743a85b27b4bcbb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:35:02 -0800 Subject: [PATCH 028/136] minor naming --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 04c31aaa5c..eed8fe8566 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,7 @@ There are several tools: For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. -If you want to make a proper PR or do containerized development (recommended), please read the full [docs/development.md](docs/development/README.md). +If you want to make a proper PR or do containerized development (recommended), please read the full [Development Guide](docs/development/README.md). ## Quickstart Development (For Hacking, not Proper Development) From 33d2863db2efc7b171af74819b4254d983aba532 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:45:43 -0800 Subject: [PATCH 029/136] add links to concepts --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index eed8fe8566..539a155c9e 100644 --- a/README.md +++ b/README.md @@ -156,11 +156,11 @@ if __name__ == "__main__": # How does DimOS work conceptually? There are several tools: -- [Modules](): The building blocks of DimOS, modules run in parallel and are defined in python as classes. Note: it is possible to use C++, Rust, Typescript, etc to write a module. -- Streams: How modules communicate, a Pub / Sub system. -- Blueprints: a way to group modules together and define their connections to each other -- RPC: how one module can call a method on another module (arguments get serialized to JSON-like binary data) -- Skills: Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- [Modules](docs/api/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. +- [Streams](docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Blueprints](docs/concepts/blueprints.md): a way to group modules together and define their connections to each other +- [RPC](docs/concepts/rpc.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Skills](docs/concepts/skills.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools # Contributing / Building From Source From fb6046a487bcbdaca49b01298d3729c2579fe230 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:48:20 -0800 Subject: [PATCH 030/136] fix broken link --- docs/api/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/configuration.md b/docs/api/configuration.md index 2977e8c3c1..b0dfe70230 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc +[Modules](docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc ```python from dataclasses import dataclass From 503425cc8395ef8065dbc317774eca0c866cdcb0 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 01:57:14 -0800 Subject: [PATCH 031/136] shorten readme --- README.md | 53 ++--------------------------------------------------- 1 file changed, 2 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 539a155c9e..2e99c79453 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin # the command needs to download the replay file (2.4gb), which takes a bit # OPTION 1: use without installing dimos -uvx --from 'dimos[base,unitree]' dimos --replay run unitree-g2 +uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 # OPTION 2: install dimos in a virtualenv uv venv && . .venv/bin/activate @@ -165,56 +165,7 @@ There are several tools: # Contributing / Building From Source -For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. - -If you want to make a proper PR or do containerized development (recommended), please read the full [Development Guide](docs/development/README.md). - -## Quickstart Development (For Hacking, not Proper Development) - -```bash - -# On Ubuntu 22.04 or 24.04 -if [ "$OSTYPE" = "linux-gnu" ]; then - sudo apt-get update - sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev -# On macOS (12.6 or newer) -elif [ "$(uname)" = "Darwin" ]; then - # install homebrew - /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" - # install dependencies - brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python -fi - -# this allows getting large files on-demand -export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git -cd dimos - -# install uv for python -curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" - -# create & activate a virtualenv (needed for dimos) -uv venv && . .venv/bin/activate - -# install everything -uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' - -# test the install (takes about 3 minutes) -uv run pytest dimos -``` - -Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: - -- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector -- **contact_graspnet_pytorch**: Required for robotic grasp prediction - -You can install them with: - -```bash -uv add git+https://github.com/openai/CLIP.git -uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git -uv add git+https://github.com/facebookresearch/detectron2.git -``` +For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](docs/development/README.md) to see the extra steps for setting up development environments. # License From 0749404b54eb876ed044c7b827d8389fcb6c73d1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:12:34 -0800 Subject: [PATCH 032/136] fix links --- README.md | 6 +++--- docs/development/README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2e99c79453..6d21039cf4 100644 --- a/README.md +++ b/README.md @@ -156,11 +156,11 @@ if __name__ == "__main__": # How does DimOS work conceptually? There are several tools: -- [Modules](docs/api/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. +- [Modules](docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. - [Streams](docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. - [Blueprints](docs/concepts/blueprints.md): a way to group modules together and define their connections to each other -- [RPC](docs/concepts/rpc.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) -- [Skills](docs/concepts/skills.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- [RPC](docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Skills](docs/concepts/blueprints.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools # Contributing / Building From Source diff --git a/docs/development/README.md b/docs/development/README.md index 4743d7f46a..0227bd4cf2 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -220,8 +220,8 @@ uv run pytest dimos ## Where is `` located? (Architecture) -* If you want to add a `dimos run ` command see [dimos_run.md](docs/development/dimos_run.md) -* For edits to manipulation see [manipulation.md](docs/development/manipulation.md) and [manipulation base](dimos/hardware/manipulators/base/component_based_architecture.md) +* If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) +* For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. From 806426e6b3dc2e9b880e850ea925a17e31ebad0f Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:13:21 -0800 Subject: [PATCH 033/136] fix links --- docs/api/configuration.md | 2 +- docs/development/README.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/api/configuration.md b/docs/api/configuration.md index b0dfe70230..dc6db2e8a8 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules](docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc +[Modules](/docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc ```python from dataclasses import dataclass diff --git a/docs/development/README.md b/docs/development/README.md index 0227bd4cf2..c5bcd30332 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -221,7 +221,7 @@ uv run pytest dimos ## Where is `` located? (Architecture) * If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) -* For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](dimos/hardware/manipulators/base/component_based_architecture.md) +* For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. @@ -254,12 +254,12 @@ You can enable a tag by selecting -m - these are configured in `./pyp # 3. How to Make a PR - Open the PR against the `dev` branch (not `main`) - **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) -- If you're writing documentation, see [writing_docs.md](docs/development/writing_docs.md) for how to write code blocks. +- If you're writing documentation, see [writing_docs.md](/docs/development/writing_docs.md) for how to write code blocks. - Less changed files = better - If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large_files.md](docs/development/large_files.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large_files.md](/docs/development/large_files.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Smaller PR's are better. From 9b4b55fa55596c55637162f3e1c03d1d95f8c87d Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:15:32 -0800 Subject: [PATCH 034/136] Update README.md --- docs/development/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/development/README.md b/docs/development/README.md index c5bcd30332..7ec429ea01 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -149,6 +149,7 @@ Install and activate [direnv](https://direnv.net/). ```sh # Install Nix curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh # make sure flakes are enabled mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" From edc2d97d4e4d05bf0e63f448b116f7191a7c8ecc Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:17:21 -0800 Subject: [PATCH 035/136] Update README.md --- docs/development/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index 7ec429ea01..9c6bcb7af9 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -186,8 +186,6 @@ uv run pytest dimos ### Quickstart -Install and activate [direnv](https://direnv.net/). - ```sh # Install Nix curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install From 3b53fe78c2495f506e905f65f77b5bc2cd23873c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:37:45 -0800 Subject: [PATCH 036/136] remove --single-branch to avoid problems with checkout --- docs/development/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index 9c6bcb7af9..7c7c2073de 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -43,7 +43,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +git clone -b main git@github.com:dimensionalOS/dimos.git cd dimos @@ -155,7 +155,7 @@ mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +git clone -b main git@github.com:dimensionalOS/dimos.git cd dimos # activate the nix .envrc @@ -194,7 +194,7 @@ mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main --single-branch git@github.com:dimensionalOS/dimos.git +git clone -b main git@github.com:dimensionalOS/dimos.git cd dimos # activate the nix development shell From 56665be06eb9810634ef1e9b6384d030fc42e8f5 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:42:22 -0800 Subject: [PATCH 037/136] add debugging --- docs/development/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/development/README.md b/docs/development/README.md index 7c7c2073de..b8dd6c2760 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -217,6 +217,16 @@ uv run pytest dimos # 2. How to Test and Modify DimOS +## Debugging + +Enable maximum logging: by adding `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1` as a prefix to the command. For example: + +```bash +DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos run unitree-go2 +``` + +This will save the rerun data to `rerun.json` in the current directory. + ## Where is `` located? (Architecture) * If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) From 082cb611751c67c0e9374eb4547aafdeb4a1c518 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 02:42:43 -0800 Subject: [PATCH 038/136] extend architecture --- docs/development/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index b8dd6c2760..0c8cbc67d3 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -4,8 +4,6 @@ 2. How to test/hack dimos (overview and our practices) 3. How to make a PR - - # 1. Setup All the tools below are optional and for your convenience. If you can get the code running on temple OS with a package manager you wrote yourself, all the power to you. @@ -233,9 +231,11 @@ This will save the rerun data to `rerun.json` in the current directory. * For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. +* `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. * `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. * `dimos/dashboard/`: Contains code related to visualization. * `dimos/protocol/`: Defines low level stuff for communication between modules. +* See `dimos/` for the remainder ## Testing From ec1316fed064a9c3b3c8548783d384d0c2176b74 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 03:05:45 -0800 Subject: [PATCH 039/136] reorganize docs --- README.md | 540 ++++-------------- docs/agents/docs/doclinks.md | 21 - docs/agents/docs/index.md | 192 ------- docs/api/configuration.md | 2 +- .../visualization.md} | 0 docs/concepts/blueprints.md | 319 +++++++++++ docs/concepts/transports.md | 6 - docs/development.md | 180 ------ docs/development/README.md | 279 +++++++++ .../assets/codeblocks_example.svg | 0 .../assets/pikchr_basic.svg | 0 .../assets/pikchr_branch.svg | 0 .../assets/pikchr_explicit.svg | 0 .../assets/pikchr_labels.svg | 0 .../assets/pikchr_sizing.svg | 0 .../depth_camera_integration.md | 0 docs/development/dimos_run.md | 79 +++ .../large_file_management.md} | 0 .../writing_docs.md} | 316 +++++++--- docs/old/ci.md | 146 ----- docs/old/jetson.MD | 72 --- docs/old/modules.md | 165 ------ docs/old/modules_CN.md | 188 ------ docs/old/ros_navigation.md | 284 --------- docs/old/running_without_devcontainer.md | 21 - docs/old/testing_stream_reply.md | 174 ------ docs/package_usage.md | 62 -- 27 files changed, 1025 insertions(+), 2021 deletions(-) delete mode 100644 docs/agents/docs/doclinks.md delete mode 100644 docs/agents/docs/index.md rename docs/{VIEWER_BACKENDS.md => api/visualization.md} (100%) create mode 100644 docs/concepts/blueprints.md delete mode 100644 docs/development.md create mode 100644 docs/development/README.md rename docs/{agents/docs => development}/assets/codeblocks_example.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_basic.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_branch.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_explicit.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_labels.svg (100%) rename docs/{agents/docs => development}/assets/pikchr_sizing.svg (100%) rename docs/{ => development}/depth_camera_integration.md (100%) create mode 100644 docs/development/dimos_run.md rename docs/{data.md => development/large_file_management.md} (100%) rename docs/{agents/docs/codeblocks.md => development/writing_docs.md} (54%) delete mode 100644 docs/old/ci.md delete mode 100644 docs/old/jetson.MD delete mode 100644 docs/old/modules.md delete mode 100644 docs/old/modules_CN.md delete mode 100644 docs/old/ros_navigation.md delete mode 100644 docs/old/running_without_devcontainer.md delete mode 100644 docs/old/testing_stream_reply.md delete mode 100644 docs/package_usage.md diff --git a/README.md b/README.md index 9a74d63aa7..6d21039cf4 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -![Screenshot 2025-02-18 at 16-31-22 DimOS Terminal](/assets/dimos_terminal.png) + + # The Dimensional Framework *The universal framework for AI-native generalist robotics* ## What is Dimensional? -Dimensional is an open-source framework for building agentive generalist robots. DimOS allows off-the-shelf Agents to call tools/functions and read sensor/state data directly from ROS. +#### Warning: This is a pre-release version -The framework enables neurosymbolic orchestration of Agents as generalized spatial reasoners/planners and Robot state/action primitives as functions. +Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. -The result: cross-embodied *"Dimensional Applications"* exceptional at generalization and robust at symbolic action execution. +# How do I get started? -## DIMOS x Unitree Go2 (OUT OF DATE) +## Installation -We are shipping a first look at the DIMOS x Unitree Go2 integration, allowing for off-the-shelf Agents() to "call" Unitree ROS2 Nodes and WebRTC action primitives, including: +#### Details / Requirements -- Navigation control primitives (move, reverse, spinLeft, spinRight, etc.) -- WebRTC control primitives (FrontPounce, FrontFlip, FrontJump, etc.) -- Camera feeds (image_raw, compressed_image, etc.) -- IMU data -- State information -- Lidar / PointCloud primitives -- Any other generic Unitree ROS2 topics - -### Features - -- **DimOS Agents** - - Agent() classes with planning, spatial reasoning, and Robot.Skill() function calling abilities. - - Integrate with any off-the-shelf hosted or local model: OpenAIAgent, ClaudeAgent, GeminiAgent 🚧, DeepSeekAgent 🚧, HuggingFaceRemoteAgent, HuggingFaceLocalAgent, etc. - - Modular agent architecture for easy extensibility and chaining of Agent output --> Subagents input. - - Agent spatial / language memory for location grounded reasoning and recall. - -- **DimOS Infrastructure** - - A reactive data streaming architecture using RxPY to manage real-time video (or other sensor input), outbound commands, and inbound robot state between the DimOS interface, Agents, and ROS2. - - Robot Command Queue to handle complex multi-step actions to Robot. - - Simulation bindings (Genesis, Isaacsim, etc.) to test your agentive application before deploying to a physical robot. - -- **DimOS Interface / Development Tools** - - Local development interface to control your robot, orchestrate agents, visualize camera/lidar streams, and debug your dimensional agentive application. - -## MacOS Installation +- Linux, tested on Ubuntu 22.04, 24.04 +- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* + - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` ```sh -# Install Nix -curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install - -# clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git - -# setup the environment (follow the prompts after nix develop) -cd dimos -nix develop - -# You should be able to follow the instructions below as well for a more manual installation -``` - ---- -## Python Installation -Tested on Ubuntu 22.04/24.04 - -```bash -sudo apt install python3-venv - -# Clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git -cd dimos - -# Create and activate virtual environment -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# Install LFS -sudo apt install git-lfs -git lfs install - -# Install torch and torchvision if not already installed -# Example CUDA 11.7, Pytorch 2.0.1 (replace with your required pytorch version if different) -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 +sudo apt-get update +sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" + +# +# NOTE!!! the first time, you're going to have an empty/black rerun window for a while +# +# the command needs to download the replay file (2.4gb), which takes a bit + +# OPTION 1: use without installing dimos +uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 + +# OPTION 2: install dimos in a virtualenv +uv venv && . .venv/bin/activate +uv pip install 'dimos[base,unitree]' +dimos --replay run unitree-go2 ``` -#### Install dependencies -```bash -# CPU only (reccomended to attempt first) -pip install -e '.[cpu,dev]' - -# CUDA install -pip install -e '.[cuda,dev]' + -# Copy and configure environment variables -cp default.env .env -``` +#### Get it working on a real physical robot! -#### Test the install -```bash -pytest -s dimos/ +```sh +export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +dimos run unitree-go2 ``` -#### Test Dimensional with a replay UnitreeGo2 stream (no robot required) -```bash -dimos --replay run unitree-go2 -``` +#### Get it working in an interactive simulation! -#### Test Dimensional with a simulated UnitreeGo2 in MuJoCo (no robot required) ```bash -pip install -e '.[sim]' export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors +# ignore the warp warnings dimos --simulation run unitree-go2 +# open http://localhost:7779/command-center in your browser to control the robot movement ``` -#### Test Dimensional with a real UnitreeGo2 over WebRTC -```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2 -``` +#### Have it controlled by AI! -#### Test Dimensional with a real UnitreeGo2 running Agents -*OpenAI / Alibaba keys required* ```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2-agentic -``` ---- - -### Agent API keys - -Full functionality will require API keys for the following: - -Requirements: -- OpenAI API key (required for all LLMAgents due to OpenAIEmbeddings) -- Claude API key (required for ClaudeAgent) -- Alibaba API key (required for Navigation skills) - -These keys can be added to your .env file or exported as environment variables. -``` export OPENAI_API_KEY= -export CLAUDE_API_KEY= -export ALIBABA_API_KEY= -``` - -### ROS2 Unitree Go2 SDK Installation - -#### System Requirements -- Ubuntu 22.04 -- ROS2 Distros: Iron, Humble, Rolling - -See [Unitree Go2 ROS2 SDK](https://github.com/dimensionalOS/go2_ros2_sdk) for additional installation instructions. - -```bash -mkdir -p ros2_ws -cd ros2_ws -git clone --recurse-submodules https://github.com/dimensionalOS/go2_ros2_sdk.git src -sudo apt install ros-$ROS_DISTRO-image-tools -sudo apt install ros-$ROS_DISTRO-vision-msgs - -sudo apt install python3-pip clang portaudio19-dev -cd src -pip install -r requirements.txt -cd .. - -# Ensure clean python install before running -source /opt/ros/$ROS_DISTRO/setup.bash -rosdep install --from-paths src --ignore-src -r -y -colcon build -``` - -### Run the test application - -#### ROS2 Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -source /opt/ros/$ROS_DISTRO/setup.bash - -export ROBOT_IP="robot_ip" #for muliple robots, just split by , -export CONN_TYPE="webrtc" -ros2 launch go2_robot_sdk robot.launch.py - -``` - -#### Python Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -python tests/run.py -``` - -#### DimOS Interface: -```bash -cd dimos/web/dimos_interface -yarn install -yarn dev # you may need to run sudo if previously built via Docker -``` - -### Project Structure (OUT OF DATE) - -``` -. -├── dimos/ -│ ├── agents/ # Agent implementations -│ │ └── memory/ # Memory systems for agents, including semantic memory -│ ├── environment/ # Environment context and sensing -│ ├── hardware/ # Hardware abstraction and interfaces -│ ├── models/ # ML model definitions and implementations -│ │ ├── Detic/ # Detic object detection model -│ │ ├── depth/ # Depth estimation models -│ │ ├── segmentation/ # Image segmentation models -│ ├── perception/ # Computer vision and sensing -│ │ ├── detection2d/ # 2D object detection -│ │ └── segmentation/ # Image segmentation pipelines -│ ├── robot/ # Robot control and hardware interface -│ │ ├── global_planner/ # Path planning at global scale -│ │ ├── local_planner/ # Local navigation planning -│ │ └── unitree/ # Unitree Go2 specific implementations -│ ├── simulation/ # Robot simulation environments -│ │ ├── genesis/ # Genesis simulation integration -│ │ └── isaac/ # NVIDIA Isaac Sim integration -│ ├── skills/ # Task-specific robot capabilities -│ │ └── rest/ # REST API based skills -│ ├── stream/ # WebRTC and data streaming -│ │ ├── audio/ # Audio streaming components -│ │ └── video_providers/ # Video streaming components -│ ├── types/ # Type definitions and interfaces -│ ├── utils/ # Utility functions and helpers -│ └── web/ # DimOS development interface and API -│ ├── dimos_interface/ # DimOS web interface -│ └── websocket_vis/ # Websocket visualizations -├── tests/ # Test files -│ ├── genesissim/ # Genesis simulator tests -│ └── isaacsim/ # Isaac Sim tests -└── docker/ # Docker configuration files - ├── agent/ # Agent service containers - ├── interface/ # Interface containers - ├── simulation/ # Simulation environment containers - └── unitree/ # Unitree robot specific containers -``` - -## Building - -### Simple DimOS Application (OUT OF DATE) - -```python -from dimos.robot.unitree.unitree_go2 import UnitreeGo2 -from dimos.robot.unitree.unitree_skills import MyUnitreeSkills -from dimos.robot.unitree.unitree_ros_control import UnitreeROSControl -from dimos.agents_deprecated.agent import OpenAIAgent - -# Initialize robot -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) - -# Initialize agent -agent = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_video_stream=robot.get_ros_video_stream(), - skills=robot.get_skills(), - system_query="Jump when you see a human! Front flip when you see a dog!", - model_name="gpt-4o" - ) - -while True: # keep process running - time.sleep(1) -``` - - -### DimOS Application with Agent chaining (OUT OF DATE) - -Let's build a simple DimOS application with Agent chaining. We define a ```planner``` as a ```PlanningAgent``` that takes in user input to devise a complex multi-step plan. This plan is passed step-by-step to an ```executor``` agent that can queue ```AbstractRobotSkill``` commands to the ```ROSCommandQueue```. - -Our reactive Pub/Sub data streaming architecture allows for chaining of ```Agent_0 --> Agent_1 --> ... --> Agent_n``` via the ```input_query_stream``` parameter in each which takes an ```Observable``` input from the previous Agent in the chain. - -**Via this method you can chain together any number of Agents() to create complex dimensional applications.** - -```python - -web_interface = RobotWebInterface(port=5555) - -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) - -# Initialize master planning agent -planner = PlanningAgent( - dev_name="UnitreePlanningAgent", - input_query_stream=web_interface.query_stream, # Takes user input from dimOS interface - skills=robot.get_skills(), - model_name="gpt-4o", - ) - -# Initialize execution agent -executor = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_query_stream=planner.get_response_observable(), # Takes planner output as input - skills=robot.get_skills(), - model_name="gpt-4o", - system_query=""" - You are a robot execution agent that can execute tasks on a virtual - robot. ONLY OUTPUT THE SKILLS TO EXECUTE. - """ - ) - -while True: # keep process running - time.sleep(1) -``` - -### Calling Action Primitives (OUT OF DATE) - -Call action primitives directly from ```Robot()``` for prototyping and testing. - -```python -robot = UnitreeGo2(ip=robot_ip,) - -# Call a Unitree WebRTC action primitive -robot.webrtc_req(api_id=1016) # "Hello" command - -# Call a ROS2 action primitive -robot.move(distance=1.0, speed=0.5) -``` - -### Creating Custom Skills (non-unitree specific) - -#### Create basic custom skills by inheriting from ```AbstractRobotSkill``` and implementing the ```__call__``` method. - -```python -class Move(AbstractRobotSkill): - distance: float = Field(...,description="Distance to reverse in meters") - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - return self._robot.move(distance=self.distance) -``` - -#### Chain together skills to create recursive skill trees - -```python -class JumpAndFlip(AbstractRobotSkill): - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - jump = Jump(robot=self._robot) - flip = Flip(robot=self._robot) - return (jump() and flip()) +dimos run unitree-go2-agentic +# open the following in a different terminal tab to tell it where to go +# Warning!: make sure to watch the bot, this is a pre-release it will run into stuff +# and get tangled/trapped +# ex: tell it to explore the room, then tell it to go to where it can see a door +humancli ``` -### Integrating Skills with Agents: Single Skills and Skill Libraries - -DimOS agents, such as `OpenAIAgent`, can be endowed with capabilities through two primary mechanisms: by providing them with individual skill classes or with comprehensive `SkillLibrary` instances. This design offers flexibility in how robot functionalities are defined and managed within your agent-based applications. - -**Agent's `skills` Parameter** - -The `skills` parameter in an agent's constructor is key to this integration: +# How do I use it as a library? -1. **A Single Skill Class**: This approach is suitable for skills that are relatively self-contained or have straightforward initialization requirements. - * You pass the skill *class itself* (e.g., `GreeterSkill`) directly to the agent's `skills` parameter. - * The agent then takes on the responsibility of instantiating this skill when it's invoked. This typically involves the agent providing necessary context to the skill's constructor (`__init__`), such as a `Robot` instance (or any other private instance variable) if the skill requires it. -2. **A `SkillLibrary` Instance**: This is the preferred method for managing a collection of skills, especially when skills have dependencies, require specific configurations, or need to share parameters. - * You first define your custom skill library by inheriting from `SkillLibrary`. Then, you create and configure an *instance* of this library (e.g., `my_lib = EntertainmentSkills(robot=robot_instance)`). - * This pre-configured `SkillLibrary` instance is then passed to the agent's `skills` parameter. The library itself manages the lifecycle and provision of its contained skills. +Simple camera activation (save this as a python file and run it): -**Examples:** +```py +from dimos.core.blueprints import autoconnect +from dimos.hardware.camera.module import CameraModule -#### 1. Using a Single Skill Class with an Agent - -First, define your skill. For instance, a `GreeterSkill` that can deliver a configurable greeting: - -```python -class GreeterSkill(AbstractSkill): - """Greats the user with a friendly message.""" # Gives the agent better context for understanding (the more detailed the better). - - greeting: str = Field(..., description="The greating message to display.") # The field needed for the calling of the function. Your agent will also pull from the description here to gain better context. - - def __init__(self, greeting_message: Optional[str] = None, **data): - super().__init__(**data) - if greeting_message: - self.greeting = greeting_message - # Any additional skill-specific initialization can go here - - def __call__(self): - super().__call__() # Call parent's method if it contains base logic - # Implement the logic for the skill - print(self.greeting) - return f"Greeting delivered: '{self.greeting}'" +if __name__ == "__main__": + autoconnect( + CameraModule.blueprint() + ).build().loop() + print("Webcam pipeline running. Press Ctrl+C to stop.") ``` -Next, register this skill *class* directly with your agent. The agent can then instantiate it, potentially with specific configurations if your agent or skill supports it (e.g., via default parameters or a more advanced setup). - -```python -agent = OpenAIAgent( - dev_name="GreetingBot", - system_query="You are a polite bot. If a user asks for a greeting, use your GreeterSkill.", - skills=GreeterSkill, # Pass the GreeterSkill CLASS - # The agent will instantiate GreeterSkill. - # If the skill had required __init__ args not provided by the agent automatically, - # this direct class passing might be insufficient without further agent logic - # or by passing a pre-configured instance (see SkillLibrary example). - # For simple skills like GreeterSkill with defaults or optional args, this works well. - model_name="gpt-4o" -) -``` -In this setup, when the `GreetingBot` agent decides to use the `GreeterSkill`, it will instantiate it. If the `GreeterSkill` were to be instantiated by the agent with a specific `greeting_message`, the agent's design would need to support passing such parameters during skill instantiation. +Write your own custom module: -#### 2. Using a `SkillLibrary` Instance with an Agent +```py +from dimos.core.blueprints import autoconnect +from dimos.core import In, Module, pSHMTransport +from dimos.core.core import rpc +from dimos.hardware.camera.module import CameraModule +from dimos.msgs.sensor_msgs import Image -Define the SkillLibrary and any skills it will manage in its collection: -```python -class MovementSkillsLibrary(SkillLibrary): - """A specialized skill library containing movement and navigation related skills.""" +from reactivex.disposable import Disposable - def __init__(self, robot=None): - super().__init__() - self._robot = robot +class Listener(Module): + # the CameraModule has an Out[Image] named "color_image" + # this module will only receive those messages if + # the names ("color_image") match, otherwise autoconnect + # will not be able to connect one module to another + color_image: In[Image] = None + grayscale_image: Out[Image] = None - def initialize_skills(self, robot=None): - """Initialize all movement skills with the robot instance.""" - if robot: - self._robot = robot + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.count = 0 - if not self._robot: - raise ValueError("Robot instance is required to initialize skills") + @rpc + def start(self) -> None: + super().start() + def callback_func(img: Image) -> None: + self.count += 1 + print(f"got frame {self.count}") + print(f"img.data.shape: {img.data.shape}") + self.grayscale_image.publish(img.to_grayscale()) - # Initialize with all movement-related skills - self.add(Navigate(robot=self._robot)) - self.add(NavigateToGoal(robot=self._robot)) - self.add(FollowHuman(robot=self._robot)) - self.add(NavigateToObject(robot=self._robot)) - self.add(GetPose(robot=self._robot)) # Position tracking skill -``` + unsubscribe_func = self.color_image.subscribe(callback_func) + # disposables will be called when the module is stopped + self._disposables.add(Disposable( + unsubscribe_func + )) -Note the addision of initialized skills added to this collection above. + @rpc + def stop(self) -> None: + super().stop() -Proceed to use this skill library in an Agent: - -Finally, in your main application code: -```python -# 1. Create an instance of your custom skill library, configured with the robot -my_movement_skills = MovementSkillsLibrary(robot=robot_instance) - -# 2. Pass this library INSTANCE to the agent -performing_agent = OpenAIAgent( - dev_name="ShowBot", - system_query="You are a show robot. Use your skills as directed.", - skills=my_movement_skills, # Pass the configured SkillLibrary INSTANCE - model_name="gpt-4o" -) +if __name__ == "__main__": + autoconnect( + Listener.blueprint(), + CameraModule.blueprint(), + ).build().loop() ``` -### Unitree Test Files -- **`tests/run_go2_ros.py`**: Tests `UnitreeROSControl(ROSControl)` initialization in `UnitreeGo2(Robot)` via direct function calls `robot.move()` and `robot.webrtc_req()` -- **`tests/simple_agent_test.py`**: Tests a simple zero-shot class `OpenAIAgent` example -- **`tests/unitree/test_webrtc_queue.py`**: Tests `ROSCommandQueue` via a 20 back-to-back WebRTC requests to the robot -- **`tests/test_planning_agent_web_interface.py`**: Tests a simple two-stage `PlanningAgent` chained to an `ExecutionAgent` with backend FastAPI interface. -- **`tests/test_unitree_agent_queries_fastapi.py`**: Tests a zero-shot `ExecutionAgent` with backend FastAPI interface. - -## Documentation - -For detailed documentation, please visit our [documentation site](#) (Coming Soon). - -## Contributing - -We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. +### Note: Many More Examples in the [Examples Folder](./examples) -## License -This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. +# How does DimOS work conceptually? -## Acknowledgments +There are several tools: +- [Modules](docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. +- [Streams](docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Blueprints](docs/concepts/blueprints.md): a way to group modules together and define their connections to each other +- [RPC](docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Skills](docs/concepts/blueprints.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools -Huge thanks to! -- The Roboverse Community and their unitree-specific help. Check out their [Discord](https://discord.gg/HEXNMCNhEh). -- @abizovnuralem for his work on the [Unitree Go2 ROS2 SDK](https://github.com/abizovnuralem/go2_ros2_sdk) we integrate with for DimOS. -- @legion1581 for his work on the [Unitree Go2 WebRTC Connect](https://github.com/legion1581/go2_webrtc_connect) from which we've pulled the ```Go2WebRTCConnection``` class and other types for seamless WebRTC-only integration with DimOS. -- @tfoldi for the webrtc_req integration via Unitree Go2 ROS2 SDK, which allows for seamless usage of Unitree WebRTC control primitives with DimOS. +# Contributing / Building From Source -## Contact +For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](docs/development/README.md) to see the extra steps for setting up development environments. -- GitHub Issues: For bug reports and feature requests -- Email: [build@dimensionalOS.com](mailto:build@dimensionalOS.com) +# License -## Known Issues -- Agent() failure to execute Nav2 action primitives (move, reverse, spinLeft, spinRight) is almost always due to the internal ROS2 collision avoidance, which will sometimes incorrectly display obstacles or be overly sensitive. Look for ```[behavior_server]: Collision Ahead - Exiting DriveOnHeading``` in the ROS logs. Reccomend restarting ROS2 or moving robot from objects to resolve. -- ```docker-compose up --build``` does not fully initialize the ROS2 environment due to ```std::bad_alloc``` errors. This will occur during continuous docker development if the ```docker-compose down``` is not run consistently before rebuilding and/or you are on a machine with less RAM, as ROS is very memory intensive. Reccomend running to clear your docker cache/images/containers with ```docker system prune``` and rebuild. +DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. diff --git a/docs/agents/docs/doclinks.md b/docs/agents/docs/doclinks.md deleted file mode 100644 index d5533c5983..0000000000 --- a/docs/agents/docs/doclinks.md +++ /dev/null @@ -1,21 +0,0 @@ -When writing or editing markdown documentation, use `doclinks` tool to resolve file references. - -Full documentation if needed: [`utils/docs/doclinks.md`](/dimos/utils/docs/doclinks.md) - -## Syntax - - -| Pattern | Example | -|-------------|-----------------------------------------------------| -| Code file | `[`service/spec.py`]()` → resolves path | -| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | -| Doc link | `[Configuration](.md)` → resolves to doc | - - -## Usage - -```bash -doclinks docs/guide.md # single file -doclinks docs/ # directory -doclinks --dry-run ... # preview only -``` diff --git a/docs/agents/docs/index.md b/docs/agents/docs/index.md deleted file mode 100644 index bec2ce79e6..0000000000 --- a/docs/agents/docs/index.md +++ /dev/null @@ -1,192 +0,0 @@ - -# Code Blocks - -**All code blocks must be executable.** -Never write illustrative/pseudo code blocks. -If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. - -After writing a code block in your markdown file, you can run it by executing -`md-babel-py run document.md` - -more information on this tool is in [codeblocks](/docs/agents/docs_agent/codeblocks.md) - - -# Code or Docs Links - -After adding a link to a doc run - -`doclinks document.md` - -### Code file references -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -After running doclinks, becomes: -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -### Symbol auto-linking -Mention a symbol on the same line to auto-link to its line number: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` - -Becomes: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` -### Doc-to-doc references -Use `.md` as the link target: -```markdown -See [Configuration](/docs/api/configuration.md) for more details. -``` - -Becomes: -```markdown -See [Configuration](/docs/concepts/configuration.md) for more details. -``` - -More information on this in [doclinks](/docs/agents/docs_agent/doclinks.md) - - -# Pikchr - -[Pikchr](https://pikchr.org/) is a diagram language from SQLite. Use it for flowcharts and architecture diagrams. - -**Important:** Always wrap pikchr blocks in `
` tags so the source is collapsed by default on GitHub. The rendered SVG stays visible outside the fold. Code blocks (Python, etc.) should NOT be folded—they're meant to be read. - -## Basic syntax - -
-diagram source - -```pikchr fold output=assets/pikchr_basic.svg -color = white -fill = none - -A: box "Step 1" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box "Step 2" rad 5px fit wid 170% ht 170% -arrow right 0.3in -C: box "Step 3" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_basic.svg) - -## Box sizing - -Use `fit` with percentage scaling to auto-size boxes with padding: - -
-diagram source - -```pikchr fold output=assets/pikchr_sizing.svg -color = white -fill = none - -# fit wid 170% ht 170% = auto-size + padding -A: box "short" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box ".subscribe()" rad 5px fit wid 170% ht 170% -arrow right 0.3in -C: box "two lines" "of text" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_sizing.svg) - -The pattern `fit wid 170% ht 170%` means: auto-size to text, then scale width by 170% and height by 170%. - -For explicit sizing (when you need consistent box sizes): - -
-diagram source - -```pikchr fold output=assets/pikchr_explicit.svg -color = white -fill = none - -A: box "Step 1" rad 5px fit wid 170% ht 170% -arrow right 0.3in -B: box "Step 2" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_explicit.svg) - -## Common settings - -Always start with: - -``` -color = white # text color -fill = none # transparent box fill -``` - -## Branching paths - -
-diagram source - -```pikchr fold output=assets/pikchr_branch.svg -color = white -fill = none - -A: box "Input" rad 5px fit wid 170% ht 170% -arrow -B: box "Process" rad 5px fit wid 170% ht 170% - -# Branch up -arrow from B.e right 0.3in then up 0.35in then right 0.3in -C: box "Path A" rad 5px fit wid 170% ht 170% - -# Branch down -arrow from B.e right 0.3in then down 0.35in then right 0.3in -D: box "Path B" rad 5px fit wid 170% ht 170% -``` - -
- - -![output](assets/pikchr_branch.svg) - -**Tip:** For tree/hierarchy diagrams, prefer left-to-right layout (root on left, children branching right). This reads more naturally and avoids awkward vertical stacking. - -## Adding labels - -
-diagram source - -```pikchr fold output=assets/pikchr_labels.svg -color = white -fill = none - -A: box "Box" rad 5px fit wid 170% ht 170% -text "label below" at (A.x, A.y - 0.4in) -``` - -
- - -![output](assets/pikchr_labels.svg) - -## Reference - -| Element | Syntax | -|---------|--------| -| Box | `box "text" rad 5px wid Xin ht Yin` | -| Arrow | `arrow right 0.3in` | -| Oval | `oval "text" wid Xin ht Yin` | -| Text | `text "label" at (X, Y)` | -| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | - -See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/api/configuration.md b/docs/api/configuration.md index 2977e8c3c1..dc6db2e8a8 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc +[Modules](/docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc ```python from dataclasses import dataclass diff --git a/docs/VIEWER_BACKENDS.md b/docs/api/visualization.md similarity index 100% rename from docs/VIEWER_BACKENDS.md rename to docs/api/visualization.md diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md new file mode 100644 index 0000000000..0a3e2ceaf5 --- /dev/null +++ b/docs/concepts/blueprints.md @@ -0,0 +1,319 @@ +# Blueprints + +Blueprints (`ModuleBlueprint`) are instructions for how to initialize a `Module`. + +You don't typically want to run a single module, so multiple blueprints are handled together in `ModuleBlueprintSet`. + +You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: + +```python +blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') +``` + +But the same thing can be acomplished more succinctly as: + +```python +connection = ConnectionModule.blueprint +``` + +Now you can create the blueprint with: + +```python +blueprint = connection('arg1', 'arg2', kwarg='value') +``` + +## Linking blueprints + +You can link multiple blueprints together with `autoconnect`: + +```python +blueprint = autoconnect( + module1(), + module2(), + module3(), +) +``` + +`blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: + +```python +expanded_blueprint = autoconnect( + blueprint, + module4(), + module5(), +) +``` + +Blueprints are frozen data classes, and `autoconnect()` always constructs an expanded blueprint so you never have to worry about changes in one affecting the other. + +### Duplicate module handling + +If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: + +```python +blueprint = autoconnect( + module_a(arg1=1), + module_b(), + module_a(arg1=2), # This one is used, the first is discarded +) +``` + +This is so you can "inherit" from one blueprint but override something you need to change. + +## How transports are linked + +Imagine you have this code: + +```python +class ModuleA(Module): + image: Out[Image] + start_explore: Out[Bool] + +class ModuleB(Module): + image: In[Image] + begin_explore: In[Bool] + +module_a = partial(create_module_blueprint, ModuleA) +module_b = partial(create_module_blueprint, ModuleB) + +autoconnect(module_a(), module_b()) +``` + +Connections are linked based on `(property_name, object_type)`. In this case `('image', Image)` will be connected between the two modules, but `begin_explore` will not be linked to `start_explore`. + +## Topic names + +By default, the name of the property is used to generate the topic name. So for `image`, the topic will be `/image`. + +The property name is used only if it's unique. If two modules have the same property name with different types, then both get a random topic such as `/SGVsbG8sIFdvcmxkI`. + +If you don't like the name you can always override it like in the next section. + +## Which transport is used? + +By default `LCMTransport` is used if the object supports `lcm_encode`. If it doesn't `pLCMTransport` is used (meaning "pickled LCM"). + +You can override transports with the `transports` method. It returns a new blueprint in which the override is set. + +```python +blueprint = autoconnect(...) +expanded_blueprint = autoconnect(blueprint, ...) +blueprint = blueprint.transports({ + ("image", Image): pSHMTransport( + "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE + ), + ("start_explore", Bool): pLCMTransport(), +}) +``` + +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. + +## Remapping connections + +Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: + +```python +class ConnectionModule(Module): + color_image: Out[Image] # Outputs on 'color_image' + +class ProcessingModule(Module): + rgb_image: In[Image] # Expects input on 'rgb_image' + +# Without remapping, these wouldn't connect automatically +# With remapping, color_image is renamed to rgb_image +blueprint = ( + autoconnect( + ConnectionModule.blueprint(), + ProcessingModule.blueprint(), + ) + .remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), + ]) +) +``` + +After remapping: +- The `color_image` output from `ConnectionModule` is treated as `rgb_image` +- It automatically connects to any module with an `rgb_image` input of type `Image` +- The topic name becomes `/rgb_image` instead of `/color_image` + +If you want to override the topic, you still have to do it manually: + +```python +blueprint +.remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), +]) +.transports({ + ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), +}) +``` + +## Overriding global configuration. + +Each module can optionally take a `global_config` option in `__init__`. E.g.: + +```python +class ModuleA(Module): + + def __init__(self, global_config: GlobalConfig | None = None): + ... +``` + +The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: + +```python +blueprint = blueprint.global_config(n_dask_workers=8) +``` + +## Calling the methods of other modules + +Imagine you have this code: + +```python +class ModuleA(Module): + + @rpc + def get_time(self) -> str: + ... + +class ModuleB(Module): + def request_the_time(self) -> None: + ... +``` + +And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. + +To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. + +```python +class ModuleB(Module): + rpc_calls: list[str] = [ + "ModuleA.get_time", + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("ModuleA.get_time") + print(get_time_rpc()) +``` + +You can also request multiple methods at a time: + +```python +method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +``` + +## Alternative RPC calls + +There is an alternative way of receiving RPC methods. It is useful when you want to perform an action at the time you receive the RPC methods. + +You can use it by defining a method like `set__`: + +```python +class ModuleB(Module): + @rpc # Note that it has to be an rpc method. + def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: + self._get_time = rpc_call + self._get_time.set_rpc(self.rpc) + + def request_the_time(self) -> None: + print(self._get_time()) +``` + +Note that `RpcCall.rpc` does not serialize, so you have to set it to the one from the module with `rpc_call.set_rpc(self.rpc)` + +## Calling an interface + +In the previous examples, you can only call methods in a module called `ModuleA`. But what if you want to deploy an alternative module in your blueprint? + +You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. + +```python +class TimeInterface(ABC): + @abstractmethod + def get_time(self): ... + +class ProperTime(TimeInterface): + def get_time(self): + return "13:00" + +class BadTime(TimeInterface): + def get_time(self): + return "01:00 PM" + + +class ModuleB(Module): + rpc_calls: list[str] = [ + "TimeInterface.get_time", # TimeInterface instead of ProperTime or BadTime + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("TimeInterface.get_time") + print(get_time_rpc()) +``` + +The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: + +```python +blueprint = autoconnect( + ProperTime.blueprint(), + # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time + ModuleB.blueprint(), +) +``` + +If both are deployed, the blueprint will throw an error because it's ambiguous. + +## Defining skills + +Skills have to be registered with `AgentSpec.register_skills(self)`. + +```python +class SomeSkill(Module): + + @skill + def some_skill(self) -> None: + ... + + @rpc + def set_AgentSpec_register_skills(self, register_skills: RpcCall) -> None: + register_skills.set_rpc(self.rpc) + register_skills(RPCClient(self, self.__class__)) + + # The agent is just interested in the `@skill` methods, so you'll need this if your class + # has things that cannot be pickled. + def __getstate__(self): + pass + def __setstate__(self, _state): + pass +``` + +Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: + +```python +class SomeSkill(SkillModule): + + @skill + def some_skill(self) -> None: + ... +``` + +## Building + +All you have to do to build a blueprint is call: + +```python +module_coordinator = blueprint.build(global_config=config) +``` + +This returns a `ModuleCoordinator` instance that manages all deployed modules. + +### Running and shutting down + +You can block the thread until it exits with: + +```python +module_coordinator.loop() +``` + +This will wait for Ctrl+C and then automatically stop all modules and clean up resources. diff --git a/docs/concepts/transports.md b/docs/concepts/transports.md index b0257114ac..fe06334fe9 100644 --- a/docs/concepts/transports.md +++ b/docs/concepts/transports.md @@ -132,12 +132,6 @@ lcm.stop() Received velocity: x=1.0, y=0.0, z=0.5 ``` -### Inspecting LCM traffic (CLI) - -- `dimos lcmspy` shows topic frequency/bandwidth stats. -- `dimos topic echo /topic` listens on typed channels like `/topic#pkg.Msg` and decodes automatically. -- `dimos topic echo /topic TypeName` is the explicit legacy form. - ## Encoder Mixins Transports can use encoder mixins to serialize messages. The `PubSubEncoderMixin` pattern wraps publish/subscribe to encode/decode automatically: diff --git a/docs/development.md b/docs/development.md deleted file mode 100644 index 838bae6fdb..0000000000 --- a/docs/development.md +++ /dev/null @@ -1,180 +0,0 @@ -# Development Environment Guide - -## Approach - -We optimise for flexibility—if your favourite editor is **notepad.exe**, you’re good to go. Everything below is tooling for convenience. - ---- - -## Dev Containers - -Dev containers give us a reproducible, container-based workspace identical to CI. - -### Why use them? - -* Consistent toolchain across all OSs. -* Unified formatting, linting and type-checking. -* Zero host-level dependencies (apart from Docker). - -### IDE quick start - -Install the *Dev Containers* plug-in for VS Code, Cursor, or your IDE of choice (you’ll likely be prompted automatically when you open our repo). - -### Shell only quick start - -Terminal within your IDE should use devcontainer transparently given you installed the plugin, but in case you want to run our shell without an IDE, you can use `./bin/dev` -(it depends on npm/node being installed) - -```sh -./bin/dev -devcontainer CLI (https://github.com/devcontainers/cli) not found. Install into repo root? (y/n): y - -added 1 package, and audited 2 packages in 8s -found 0 vulnerabilities - -[1 ms] @devcontainers/cli 0.76.0. Node.js v20.19.0. linux 6.12.27-amd64 x64. -[4838 ms] Start: Run: docker start f0355b6574d9bd277d6eb613e1dc32e3bc18e7493e5b170e335d0e403578bcdb -[5299 ms] f0355b6574d9bd277d6eb613e1dc32e3bc18e7493e5b170e335d0e403578bcdb -{"outcome":"success","containerId":"f0355b6574d9bd277d6eb613e1dc32e3bc18e7493e5b170e335d0e403578bcdb","remoteUser":"root","remoteWorkspaceFolder":"/workspaces/dimos"} - - ██████╗ ██╗███╗ ███╗███████╗███╗ ██╗███████╗██╗ ██████╗ ███╗ ██╗ █████╗ ██╗ - ██╔══██╗██║████╗ ████║██╔════╝████╗ ██║██╔════╝██║██╔═══██╗████╗ ██║██╔══██╗██║ - ██║ ██║██║██╔████╔██║█████╗ ██╔██╗ ██║███████╗██║██║ ██║██╔██╗ ██║███████║██║ - ██║ ██║██║██║╚██╔╝██║██╔══╝ ██║╚██╗██║╚════██║██║██║ ██║██║╚██╗██║██╔══██║██║ - ██████╔╝██║██║ ╚═╝ ██║███████╗██║ ╚████║███████║██║╚██████╔╝██║ ╚████║██║ ██║███████╗ - ╚═════╝ ╚═╝╚═╝ ╚═╝╚══════╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═════╝ ╚═╝ ╚═══╝╚═╝ ╚═╝╚══════╝ - - v_unknown:unknown | Wed May 28 09:23:33 PM UTC 2025 - -root@dimos:/workspaces/dimos # -``` - -The script will: - -* Offer to npm install `@devcontainers/cli` locally (if not available globally) on first run. -* Pull `ghcr.io/dimensionalos/dev:dev` if not present (external contributors: we plan to mirror to Docker Hub). - -You’ll land in the workspace as **root** with all project tooling available. - -## Pre-Commit Hooks - -We use [pre-commit](https://pre-commit.com) (config in `.pre-commit-config.yaml`) to enforce formatting, licence headers, EOLs, LFS checks, etc. Hooks run in **milliseconds**. -Hooks also run in CI; any auto-fixes are committed back to your PR, so local installation is optional — but gives faster feedback. - -```sh -CRLF end-lines checker...................................................Passed -CRLF end-lines remover...................................................Passed -Insert license in comments...............................................Passed -ruff format..............................................................Passed -check for case conflicts.................................................Passed -check json...............................................................Passed -check toml...............................................................Passed -check yaml...............................................................Passed -format json..............................................................Passed -LFS data.................................................................Passed - -``` -Given your editor uses ruff via devcontainers (which it should) actual auto-commit hook won't ever reformat your code - IDE will have already done this. - -### Running hooks manually - -Given your editor uses git via devcontainers (which it should) auto-commit hooks will run automatically, this is in case you want to run them manually. - -Inside the dev container (Your IDE will likely run this transparently for each commit if using devcontainer plugin): - -```sh -pre-commit run --all-files -``` - -### Installing pre-commit on your host - -```sh -apt install pre-commit # or brew install pre-commit -pre-commit install # install git hook -pre-commit run --all-files -``` - - ---- - -## Testing - -All tests run with **pytest** inside the dev container, ensuring local results match CI. - -### Basic usage - -```sh -./bin/dev # start container -pytest # run all tests beneath the current directory -``` - -Depending on which dir you are in, only tests from that dir will run, which is convinient when developing - you can frequently validate your feature tree. - -Your vibe coding agent will know to use these tests via the devcontainer so it can validate it's work. - - -#### Useful options - -| Purpose | Command | -| -------------------------- | ----------------------- | -| Show `print()` output | `pytest -s` | -| Filter by name substring | `pytest -k ""` | -| Run tests with a given tag | `pytest -m ` | - - -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) - -You can enable a tag by selecting -m - these are configured in `./pyproject.toml` - -```sh -root@dimos:/workspaces/dimos/dimos # pytest -sm vis -k my_visualization -... -``` - -Classic development run within a subtree: - -```sh -./bin/dev - -... container init ... - -root@dimos:/workspaces/dimos # cd dimos/robot/unitree_webrtc/ -root@dimos:/workspaces/dimos/dimos/robot/unitree_webrtc # pytest -collected 27 items / 22 deselected / 5 selected - -type/test_map.py::test_robot_mapping PASSED -type/test_timeseries.py::test_repr PASSED -type/test_timeseries.py::test_equals PASSED -type/test_timeseries.py::test_range PASSED -type/test_timeseries.py::test_duration PASSED - -``` - -Showing prints: - -```sh -root@dimos:/workspaces/dimos/dimos/robot/unitree_webrtc/type # pytest -s test_odometry.py -test_odometry.py::test_odometry_conversion_and_count Odom ts(2025-05-30 13:52:03) pos(→ Vector Vector([0.432199 0.108042 0.316589])), rot(↑ Vector Vector([ 7.7200000e-04 -9.1280000e-03 3.006 -8621e+00])) yaw(172.3°) -Odom ts(2025-05-30 13:52:03) pos(→ Vector Vector([0.433629 0.105965 0.316143])), rot(↑ Vector Vector([ 0.003814 -0.006436 2.99591235])) yaw(171.7°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.434459 0.104739 0.314794])), rot(↗ Vector Vector([ 0.005558 -0.004183 3.00068456])) yaw(171.9°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.435621 0.101699 0.315852])), rot(↑ Vector Vector([ 0.005391 -0.006002 3.00246893])) yaw(172.0°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.436457 0.09857 0.315254])), rot(↑ Vector Vector([ 0.003358 -0.006916 3.00347172])) yaw(172.1°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.435535 0.097022 0.314399])), rot(↑ Vector Vector([ 1.88300000e-03 -8.17800000e-03 3.00573432e+00])) yaw(172.2°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.433739 0.097553 0.313479])), rot(↑ Vector Vector([ 8.10000000e-05 -8.71700000e-03 3.00729616e+00])) yaw(172.3°) -Odom ts(2025-05-30 13:52:04) pos(→ Vector Vector([0.430924 0.09859 0.31322 ])), rot(↑ Vector Vector([ 1.84000000e-04 -9.68700000e-03 3.00945623e+00])) yaw(172.4°) -... etc -``` ---- - -## Cheatsheet - -| Action | Command | -| --------------------------- | ---------------------------- | -| Enter dev container | `./bin/dev` | -| Run all pre-commit hooks | `pre-commit run --all-files` | -| Install hooks in local repo | `pre-commit install` | -| Run tests in current path | `pytest` | -| Filter tests by name | `pytest -k ""` | -| Enable stdout in tests | `pytest -s` | -| Run tagged tests | `pytest -m ` | diff --git a/docs/development/README.md b/docs/development/README.md new file mode 100644 index 0000000000..286038caee --- /dev/null +++ b/docs/development/README.md @@ -0,0 +1,279 @@ +# Development Environment Guide + +1. How to setup your system (pick one: system install, nix flake + direnv, pure nix flake) +2. How to hack on DimOS +3. How to make a PR + +# 1. Setup + +All the tools below are optional and for your convenience. If you can get the code running on temple OS with a package manager you wrote yourself, all the power to you. + +--- + +## Setup Option A: System Install + +### Why pick this setup? (pros/cons/when-to-use) + +* Downside: not reliable, mutates your global system, causing (and receiving) side effects +* Upside: Often good for a quick hack or exploring +* Upside: Sometimes easier for CUDA/GPU acceleration +* Use when: you understand system package management (arch linux user) or you don't care about making changes to your system + +### How to setup DimOS + +```bash +# System dependencies + +# On Ubuntu 22.04 or 24.04 +if [ "$OSTYPE" = "linux-gnu" ]; then + sudo apt-get update + sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev pre-commit +# On macOS (12.6 or newer) +elif [ "$(uname)" = "Darwin" ]; then + # install homebrew + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + # install dependencies + brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python pre-commit +fi + +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + + +# create & activate a virtualenv (needed for dimos) +uv venv && . .venv/bin/activate + +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' + +# setup pre-commit +pre-commit install + +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: + +- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector +- **contact_graspnet_pytorch**: Required for robotic grasp prediction + +You can install them with: + +```bash +uv add git+https://github.com/openai/CLIP.git +uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git +uv add git+https://github.com/facebookresearch/detectron2.git +``` + + + + +## Setup Option B: Nix Flake + direnv + +### Why pick this setup? (pros/cons/when-to-use) + +* Upside: Faster and more reliable than Dev Containers (no emulation) +* Upside: Nearly as isolated as Docker, but has full hardware access (CUDA, Webcam, networking) +* Downside: Not hard, but you need to install/understand [direnv](https://direnv.net/) (which you probably should do anyway) +* Downside: Nix is not user-friendly (IDE integration is not as good as Dev Containers) +* Use when: you need reliability and don't mind a one-time startup delay + +### Quickstart + +Install and activate [direnv](https://direnv.net/). + +```sh +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix .envrc +cp .envrc.nix .envrc +# this is going to take a while +direnv allow +direnv reload +direnv status + +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +## Setup Option C: Nix Flake - Isolated/Reliable + +### Why pick this setup? (pros/cons/when-to-use) + +* Use when: you need absolute reliability (use this if you want it to work first try) and don't mind a startup delay +* Upside: Doesn't need direnv, and has most of the other benefits of Nix +* Downside: Have to manually enter the environment (like `./venv/bin/activate` but slower) +* Upside: If you're you're using a basic shell, you'll get a nicely customized shell +* Downside: If you have hyper-customized your shell (fish, riced zsh, etc), you'll have to deal with someone else's preferences +* Downside: Your vibe coding agent will basically be unable to run tests for you (they don't understand how to enter the environment) + +### Quickstart + +```sh +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix development shell +nix develop '.#isolated' +``` + +Once inside the shell, run: + +```sh +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +
+
+ +# 2. How to Hack on DimOS + +## Debugging + +Enable maximum logging: by adding `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1` as a prefix to the command. For example: + +```bash +DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos run unitree-go2 +``` + +This will save the rerun data to `rerun.json` in the current directory. + +## Where is `` located? (Architecture) + +* If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) +* If you want to add a camera driver see [depth_camera_integration.md](/docs/development/depth_camera_integration.md) +* For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) +* `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. +* `dimos/robot/`: Robot-specific modules live here. +* `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. +* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. +* `dimos/dashboard/`: Contains code related to visualization. +* `dimos/protocol/`: Defines low level stuff for communication between modules. +* See `dimos/` for the remainder + +## Testing + +We use both pytest and manual testing. + +```sh +pytest # run all tests at or below the current directory +``` + +### Testing Cheatsheet + +| Action | Command | +| --------------------------- | ---------------------------- | +| Run tests in current path | `pytest` | +| Filter tests by name | `pytest -k ""` | +| Enable stdout in tests | `pytest -s` | +| Run tagged tests | `pytest -m ` | +| Run all pre-commit hooks | `pre-commit run --all-files` | + +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) + +You can enable a tag by selecting -m - these are configured in `./pyproject.toml` + + +# 3. How to Make a PR +- Open the PR against the `dev` branch (not `main`) +- **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) +- If you're writing documentation, see [writing_docs.md](/docs/development/writing_docs.md) for how to write code blocks. +- Less changed files = better +- If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. +- We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). +- Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. +- If you have large (>500kb) files, see [large_files.md](/docs/development/large_files.md) for how to store and load them (don't just commit them). +- So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. +- If you're a new hire at DimOS: + - Smaller PR's are better. + - Only open a PR when you're okay with us spending AI tokens reviewing it (don't just open a trash PR and then fix it, wait till the code is mostly done) + - If there are 3 highly-intertwined bugs, make 3 PRs, not 1 PR. Yes it is more dev work, but review time is the bottleneck (not dev time). One line PR's are the easiest thing to review. + - When the AI (currently Greptile) comments on the code, respond. Sometimes Greptile is dumb as rocks but, as a reviewer, it's nice to see a finished conversation. + - Did we mention smaller PR's are better? diff --git a/docs/agents/docs/assets/codeblocks_example.svg b/docs/development/assets/codeblocks_example.svg similarity index 100% rename from docs/agents/docs/assets/codeblocks_example.svg rename to docs/development/assets/codeblocks_example.svg diff --git a/docs/agents/docs/assets/pikchr_basic.svg b/docs/development/assets/pikchr_basic.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_basic.svg rename to docs/development/assets/pikchr_basic.svg diff --git a/docs/agents/docs/assets/pikchr_branch.svg b/docs/development/assets/pikchr_branch.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_branch.svg rename to docs/development/assets/pikchr_branch.svg diff --git a/docs/agents/docs/assets/pikchr_explicit.svg b/docs/development/assets/pikchr_explicit.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_explicit.svg rename to docs/development/assets/pikchr_explicit.svg diff --git a/docs/agents/docs/assets/pikchr_labels.svg b/docs/development/assets/pikchr_labels.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_labels.svg rename to docs/development/assets/pikchr_labels.svg diff --git a/docs/agents/docs/assets/pikchr_sizing.svg b/docs/development/assets/pikchr_sizing.svg similarity index 100% rename from docs/agents/docs/assets/pikchr_sizing.svg rename to docs/development/assets/pikchr_sizing.svg diff --git a/docs/depth_camera_integration.md b/docs/development/depth_camera_integration.md similarity index 100% rename from docs/depth_camera_integration.md rename to docs/development/depth_camera_integration.md diff --git a/docs/development/dimos_run.md b/docs/development/dimos_run.md new file mode 100644 index 0000000000..9fb0b5845e --- /dev/null +++ b/docs/development/dimos_run.md @@ -0,0 +1,79 @@ +# DimOS Run + +#### Warning: If you just want to run a blueprint you don't need to add it to `dimos run`: + +`your_code.py` +```py +from dimos.robot.unitree_webrtc.unitree_go2_blueprints import basic as example_blueprint + +if __name__ == "__main__": + example_blueprint.build().loop() +``` + +```sh +python ./your_code.py +``` + +## Usage + +For example, to run the standard Unitree Go2 blueprint run: + +```bash +dimos run unitree-go2 +``` + +For the one with agents run: + +```bash +dimos run unitree-go2-agentic +``` + +You can dynamically connect additional modules. For example: + +```bash +dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill +``` + +## Adding your own + +Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: + +```python +all_blueprints = { + "unitree-go2": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:standard", + "unitree-go2-agentic": "dimos.robot.unitree_webrtc.unitree_go2_blueprints:agentic", + ... +} +``` + +(They are defined as imports to avoid triggering unrelated imports.) + +## `GlobalConfig` + +This tool also initializes the global config and passes it to the blueprint. + +`GlobalConfig` contains configuration options that are useful across many modules. For example: + +```python +class GlobalConfig(BaseSettings): + robot_ip: str | None = None + simulation: bool = False + replay: bool = False + n_dask_workers: int = 2 +``` + +Configuration values can be set from multiple places in order of precedence (later entries override earlier ones): + +- Default value defined on GlobalConfig. (`simulation = False`) +- Value defined in `.env` (`SIMULATION=true`) +- Value in the environment variable (`SIMULATION=true`) +- Value defined on the blueprint (`blueprint.global_config(simulation=True)`) +- Value coming from the CLI (`--simulation` or `--no-simulation`) + +For environment variables/`.env` values, you have to prefix the name with `DIMOS_`. + +For the command line, you call it like this: + +```bash +dimos --simulation run unitree-go2 +``` diff --git a/docs/data.md b/docs/development/large_file_management.md similarity index 100% rename from docs/data.md rename to docs/development/large_file_management.md diff --git a/docs/agents/docs/codeblocks.md b/docs/development/writing_docs.md similarity index 54% rename from docs/agents/docs/codeblocks.md rename to docs/development/writing_docs.md index 323f1c0c50..b9ae0b27df 100644 --- a/docs/agents/docs/codeblocks.md +++ b/docs/development/writing_docs.md @@ -1,16 +1,232 @@ -# Executable Code Blocks +# Need-to-know Things + +1. How we resolve file references (linking) +2. How we make of our code blocks executable and test them. +3. How we make diagrams + + +
+
+ +# 1. Use Doclinks to Resolve file references + +## Syntax + + +| Pattern | Example | +|-------------|-----------------------------------------------------| +| Code file | `[`service/spec.py`]()` → resolves path | +| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | +| Doc link | `[Configuration](.md)` → resolves to doc | + + +## Usage + +```bash +doclinks docs/guide.md # single file +doclinks docs/ # directory +doclinks --dry-run ... # preview only +``` + +## Full Documentation + +
+Click to see full documentation + +## What it does + +When writing docs, you can use placeholder links like: + + +```markdown +See [`service/spec.py`]() for the implementation. +``` + + +Running `doclinks` resolves these to actual paths: + + +```markdown +See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. +``` + + +## Features + + +- **Code file links**: `[`filename.py`]()` resolves to the file's path +- **Symbol line linking**: If another backticked term appears on the same line, it finds that symbol in the file and adds `#L`: + ```markdown + See `Configurable` in [`config.py`]() + → [`config.py`](/path/config.py#L42) + ``` +- **Doc-to-doc links**: `[Modules](.md)` resolves to `modules.md` or `modules/index.md` + +- **Multiple link modes**: absolute, relative, or GitHub URLs +- **Watch mode**: Automatically re-process on file changes +- **Ignore regions**: Skip sections with `` comments + +## Usage + +```bash +# Process a single file +doclinks docs/guide.md + +# Process a directory recursively +doclinks docs/ + +# Relative links (from doc location) +doclinks --link-mode relative docs/ + +# GitHub links +doclinks --link-mode github \ + --github-url https://github.com/org/repo docs/ + +# Dry run (preview changes) +doclinks --dry-run docs/ + +# CI check (exit 1 if changes needed) +doclinks --check docs/ + +# Watch mode (auto-update on changes) +doclinks --watch docs/ +``` + +## Options + +| Option | Description | +|--------------------|-------------------------------------------------| +| `--root PATH` | Repository root (default: auto-detect git root) | +| `--link-mode MODE` | `absolute` (default), `relative`, or `github` | +| `--github-url URL` | Base GitHub URL (required for github mode) | +| `--github-ref REF` | Branch/ref for GitHub links (default: `main`) | +| `--dry-run` | Show changes without modifying files | +| `--check` | Exit with error if changes needed (for CI) | +| `--watch` | Watch for changes and re-process | + +## Link patterns + + +| Pattern | Description | +|----------------------|------------------------------------------------| +| `[`file.py`]()` | Code file reference (empty or any link) | +| `[`path/file.py`]()` | Code file with partial path for disambiguation | +| `[`file.py`](#L42)` | Preserves existing line fragments | +| `[Doc Name](.md)` | Doc-to-doc link (resolves by name) | + + +## How resolution works + +The tool builds an index of all files in the repo. For `/dimos/protocol/service/spec.py`, it creates lookup entries for: + +- `spec.py` +- `service/spec.py` +- `protocol/service/spec.py` +- `dimos/protocol/service/spec.py` + +Use longer paths when multiple files share the same name. + +
+ + + +
+
+ +# 2. Code Blocks Must Be Executable We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. ## Golden Rule -**All code blocks must be executable.** Never write illustrative/pseudo code blocks. If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. +**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. + +## Installation + +
+Click to see full installation instructions + +### Nix (recommended) + +```sh skip +# (assuming you have nix) + +# Run directly from GitHub +nix run github:leshy/md-babel-py -- run README.md --stdout + +# run locally +nix run . -- run README.md --stdout +``` + +### Docker + +```sh skip +# Pull from Docker Hub +docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout + +# Or build locally via Nix +nix build .#docker # builds tarball to ./result +docker load < result # loads image from tarball +docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout +``` + +### pipx + +```sh skip +pipx install md-babel-py +# or: uv pip install md-babel-py +md-babel-py run README.md --stdout +``` + +If not using nix or docker, evaluators require system dependencies: + +| Language | System packages | +|-----------|-----------------------------| +| python | python3 | +| node | nodejs | +| dot | graphviz | +| asymptote | asymptote, texlive, dvisvgm | +| pikchr | pikchr | +| openscad | openscad, xvfb, imagemagick | +| diagon | diagon | + +```sh skip +# Arch Linux +sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick + +# Debian/Ubuntu +sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad +``` + +Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. + +## Usage + +```sh skip +# Edit file in-place +md-babel-py run document.md + +# Output to separate file +md-babel-py run document.md --output result.md + +# Print to stdout +md-babel-py run document.md --stdout + +# Only run specific languages +md-babel-py run document.md --lang python,sh + +# Dry run - show what would execute +md-babel-py run document.md --dry-run +``` + +
+ ## Running ```sh skip -md-babel-py run document.md # edit in-place -md-babel-py run document.md --stdout # preview to stdout +md-babel-py run document.md # edit in-place +md-babel-py run document.md --stdout # preview to stdout md-babel-py run document.md --dry-run # show what would run ``` @@ -113,6 +329,19 @@ plt.savefig('{output}', transparent=True) ![output](assets/matplotlib-demo.svg) + + + + + +
+
+ +# 3. Diagrams + +We have many diagraming tools. View source code of this page to see examples. + + ### Pikchr SQLite's diagram language: @@ -240,75 +469,14 @@ A -> C └───┘ ``` -## Install +## Reference -### Nix (recommended) +| Element | Syntax | +|---------|--------| +| Box | `box "text" rad 5px wid Xin ht Yin` | +| Arrow | `arrow right 0.3in` | +| Oval | `oval "text" wid Xin ht Yin` | +| Text | `text "label" at (X, Y)` | +| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | -```sh skip -# Run directly from GitHub -nix run github:leshy/md-babel-py -- run README.md --stdout - -# Or clone and run locally -nix run . -- run README.md --stdout -``` - -### Docker - -```sh skip -# Pull from Docker Hub -docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout - -# Or build locally via Nix -nix build .#docker # builds tarball to ./result -docker load < result # loads image from tarball -docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout -``` - -### pipx - -```sh skip -pipx install md-babel-py -# or: uv pip install md-babel-py -md-babel-py run README.md --stdout -``` - -If not using nix or docker, evaluators require system dependencies: - -| Language | System packages | -|-----------|-----------------------------| -| python | python3 | -| node | nodejs | -| dot | graphviz | -| asymptote | asymptote, texlive, dvisvgm | -| pikchr | pikchr | -| openscad | openscad, xvfb, imagemagick | -| diagon | diagon | - -```sh skip -# Arch Linux -sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick - -# Debian/Ubuntu -sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad -``` - -Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. - -## Usage - -```sh skip -# Edit file in-place -md-babel-py run document.md - -# Output to separate file -md-babel-py run document.md --output result.md - -# Print to stdout -md-babel-py run document.md --stdout - -# Only run specific languages -md-babel-py run document.md --lang python,sh - -# Dry run - show what would execute -md-babel-py run document.md --dry-run -``` +See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/old/ci.md b/docs/old/ci.md deleted file mode 100644 index ac9b11115a..0000000000 --- a/docs/old/ci.md +++ /dev/null @@ -1,146 +0,0 @@ -# Continuous Integration Guide - -> *If you are ******not****** editing CI-related files, you can safely ignore this document.* - -Our GitHub Actions pipeline lives in **`.github/workflows/`** and is split into three top-level workflows: - -| Workflow | File | Purpose | -| ----------- | ------------- | -------------------------------------------------------------------- | -| **cleanup** | `cleanup.yml` | Auto-formats code with *pre-commit* and pushes fixes to your branch. | -| **docker** | `docker.yml` | Builds (and caches) our Docker image hierarchy. | -| **tests** | `tests.yml` | Pulls the *dev* image and runs the test suite. | - ---- - -## `cleanup.yml` - -* Checks out the branch. -* Executes **pre-commit** hooks. -* If hooks modify files, commits and pushes the changes back to the same branch. - -> This guarantees consistent formatting even if the developer has not installed pre-commit locally. - ---- - -## `tests.yml` - -* Pulls the pre-built **dev** container image. -* Executes: - -```bash -pytest -``` - -That’s it—making the job trivial to reproduce locally via: - -```bash -./bin/dev # enter container -pytest # run tests -``` - ---- - -## `docker.yml` - -### Objectives - -1. **Layered images**: each image builds on its parent, enabling parallel builds once dependencies are ready. -2. **Speed**: build children as soon as parents finish; leverage aggressive caching. -3. **Minimal work**: skip images whose context hasn’t changed. - -### Current hierarchy - - -``` - ┌──────┐ - │ubuntu│ - └┬────┬┘ - ┌▽──┐┌▽───────┐ - │ros││python │ - └┬──┘└───────┬┘ - ┌▽─────────┐┌▽──┐ - │ros-python││dev│ - └┬─────────┘└───┘ - ┌▽──────┐ - │ros-dev│ - └───────┘ -``` - -* ghcr.io/dimensionalos/ros:dev -* ghcr.io/dimensionalos/python:dev -* ghcr.io/dimensionalos/ros-python:dev -* ghcr.io/dimensionalos/ros-dev:dev -* ghcr.io/dimensionalos/dev:dev - -> **Note**: The diagram shows only currently active images; the system is extensible—new combinations are possible, builds can be run per branch and as parallel as possible - - -``` - ┌──────┐ - │ubuntu│ - └┬────┬┘ - ┌▽──┐┌▽────────────────────────┐ - │ros││python │ - └┬──┘└───────────────────┬────┬┘ - ┌▽─────────────────────┐┌▽──┐┌▽──────┐ - │ros-python ││dev││unitree│ - └┬────────┬───────────┬┘└───┘└───────┘ - ┌▽──────┐┌▽─────────┐┌▽──────────┐ - │ros-dev││ros-jetson││ros-unitree│ - └───────┘└──────────┘└───────────┘ -``` - -### Branch-aware tagging - -When a branch triggers a build: - -* Only images whose context changed are rebuilt. -* New images receive the tag `:`. -* Unchanged parents are pulled from the registry, e.g. - -given we made python requirements.txt changes, but no ros changes, image dep graph would look like this: - -``` -ghcr.io/dimensionalos/ros:dev → ghcr.io/dimensionalos/ros-python:my_branch → ghcr.io/dimensionalos/dev:my_branch -``` - -### Job matrix & the **check-changes** step - -To decide what to build we run a `check-changes` job that compares the diff against path filters: - -```yaml -filters: | - ros: - - .github/workflows/_docker-build-template.yml - - .github/workflows/docker.yml - - docker/base-ros/** - - python: - - docker/base-python/** - - requirements*.txt - - dev: - - docker/dev/** -``` - -This populates a build matrix (ros, python, dev) with `true/false` flags. - -### The dependency execution issue - -Ideally a child job (e.g. **ros-python**) should depend on both: - -* **check-changes** (to know if it *should* run) -* Its **parent image job** (to wait for the artifact) - -GitHub Actions can’t express “run only if *both* conditions are true *and* the parent job wasn’t skipped”. - -We are using `needs: [check-changes, ros]` to ensure the job runs after the ros build, but if ros build has been skipped we need `if: always()` to ensure that the build runs anyway. -Adding `always` for some reason completely breaks the conditional check, we cannot have OR, AND operators, it just makes the job _always_ run, which means we build python even if we don't need to. - -This is unfortunate as the build takes ~30 min first time (a few minutes afterwards thanks to caching) and I've spent a lot of time on this, lots of viable seeming options didn't pan out and probably we need to completely rewrite and own the actions runner and not depend on github structure at all. Single job called `CI` or something, within our custom docker image. - ---- - -## `run-tests` (job inside `docker.yml`) - -After all requested images are built, this job triggers **tests.yml**, passing the freshly created *dev* image tag so the suite runs against the branch-specific environment. diff --git a/docs/old/jetson.MD b/docs/old/jetson.MD deleted file mode 100644 index a4d06e3255..0000000000 --- a/docs/old/jetson.MD +++ /dev/null @@ -1,72 +0,0 @@ -# DimOS Jetson Setup Instructions -Tested on Jetpack 6.2, CUDA 12.6 - -## Required system dependencies -`sudo apt install portaudio19-dev python3-pyaudio` - -## Installing cuSPARSELt -https://ninjalabo.ai/blogs/jetson_pytorch.html - -```bash -wget https://developer.download.nvidia.com/compute/cusparselt/0.7.0/local_installers/cusparselt-local-tegra-repo-ubuntu2204-0.7.0_1.0-1_arm64.deb -sudo dpkg -i cusparselt-local-tegra-repo-ubuntu2204-0.7.0_1.0-1_arm64.deb -sudo cp /var/cusparselt-local-tegra-repo-ubuntu2204-0.7.0/cusparselt-*-keyring.gpg /usr/share/keyrings/ -sudo apt-get update -sudo apt-get install libcusparselt0 libcusparselt-dev -ldconfig -``` -## Install Torch and Torchvision wheels - -Enter virtualenv -```bash -python3 -m venv venv -source venv/bin/activate -``` - -Wheels for jp6/cu126 -https://pypi.jetson-ai-lab.io/jp6/cu126 - -Check compatibility: -https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform-release-notes/pytorch-jetson-rel.html - -### Working torch wheel tested on Jetpack 6.2, CUDA 12.6 -`pip install --no-cache https://developer.download.nvidia.com/compute/redist/jp/v61/pytorch/torch-2.5.0a0+872d972e41.nv24.08.17622132-cp310-cp310-linux_aarch64.whl` - -### Install torchvision from source: -```bash -# Set version by checking above torchvision<-->torch compatibility - -# We use 0.20.0 -export VERSION=20 - -sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libopenblas-dev libavcodec-dev libavformat-dev libswscale-dev -git clone --branch release/0.$VERSION https://github.com/pytorch/vision torchvision -cd torchvision -export BUILD_VERSION=0.$VERSION.0 -python3 setup.py install --user # remove --user if installing in virtualenv -``` - -### Verify success: -```bash -$ python3 -import torch -print(torch.__version__) -print('CUDA available: ' + str(torch.cuda.is_available())) # Should be True -print('cuDNN version: ' + str(torch.backends.cudnn.version())) -a = torch.cuda.FloatTensor(2).zero_() -print('Tensor a = ' + str(a)) -b = torch.randn(2).cuda() -print('Tensor b = ' + str(b)) -c = a + b -print('Tensor c = ' + str(c)) - -$ python3 -import torchvision -print(torchvision.__version__) -``` - -## Install Onnxruntime-gpu - -Find pre-build wheels here for your specific JP/CUDA version: https://pypi.jetson-ai-lab.io/jp6 - -`pip install https://pypi.jetson-ai-lab.io/jp6/cu126/+f/4eb/e6a8902dc7708/onnxruntime_gpu-1.23.0-cp310-cp310-linux_aarch64.whl#sha256=4ebe6a8902dc7708434b2e1541b3fe629ebf434e16ab5537d1d6a622b42c622b` diff --git a/docs/old/modules.md b/docs/old/modules.md deleted file mode 100644 index 9cdbf586ac..0000000000 --- a/docs/old/modules.md +++ /dev/null @@ -1,165 +0,0 @@ -# Dimensional Modules - -The DimOS Module system enables distributed, multiprocess robotics applications using Dask for compute distribution and LCM (Lightweight Communications and Marshalling) for high-performance IPC. - -## Core Concepts - -### 1. Module Definition -Modules are Python classes that inherit from `dimos.core.Module` and define inputs, outputs, and RPC methods: - -```python -from dimos.core import Module, In, Out, rpc -from dimos.msgs.geometry_msgs import Vector3 - -class MyModule(Module): - # Declare inputs/outputs as class attributes initialized to None - data_in: In[Vector3] = None - data_out: Out[Vector3] = None - - def __init__(): - # Call parent Module init - super().__init__() - - @rpc - def remote_method(self, param): - """Methods decorated with @rpc can be called remotely""" - return param * 2 -``` - -### 2. Module Deployment -Modules are deployed across Dask workers using the `dimos.deploy()` method: - -```python -from dimos import core - -# Start Dask cluster with N workers -dimos = core.start(4) - -# Deploying modules allows for passing initialization parameters. -# In this case param1 and param2 are passed into Module init -module = dimos.deploy(Module, param1="value1", param2=123) -``` - -### 3. Stream Connections -Modules communicate via reactive streams using LCM transport: - -```python -# Configure LCM transport for outputs -module1.data_out.transport = core.LCMTransport("/topic_name", MessageType) - -# Connect module inputs to outputs -module2.data_in.connect(module1.data_out) - -# Access the underlying Observable stream -stream = module1.data_out.observable() -stream.subscribe(lambda msg: print(f"Received: {msg}")) -``` - -### 4. Module Lifecycle -```python -# Start modules to begin processing -module.start() # Calls the @rpc start() method if defined - -# Inspect module I/O configuration -print(module.io().result()) # Shows inputs, outputs, and RPC methods - -# Clean shutdown -dimos.shutdown() -``` - -## Real-World Example: Robot Control System - -```python -# Connection module wraps robot hardware/simulation -connection = dimos.deploy(ConnectionModule, ip=robot_ip) -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) -connection.video.transport = core.LCMTransport("/video", Image) - -# Perception module processes sensor data -perception = dimos.deploy(PersonTrackingStream, camera_intrinsics=[...]) -perception.video.connect(connection.video) -perception.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# Start processing -connection.start() -perception.start() - -# Enable tracking via RPC -perception.enable_tracking() - -# Get latest tracking data -data = perception.get_tracking_data() -``` - -## LCM Transport Configuration - -```python -# Standard LCM transport for simple types like lidar -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) - -# Pickle-based transport for complex Python objects / dictionaries -connection.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# Auto-configure LCM system buffers (required in containers) -from dimos.protocol import pubsub -pubsub.lcm.autoconf() -``` - -This architecture enables building complex robotic systems as composable, distributed modules that communicate efficiently via streams and RPC, scaling from single machines to clusters. - -# Dimensional Install -## Python Installation (Ubuntu 22.04) - -```bash -sudo apt install python3-venv - -# Clone the repository (dev branch, no submodules) -git clone -b dev https://github.com/dimensionalOS/dimos.git -cd dimos - -# Create and activate virtual environment -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# Install torch and torchvision if not already installed -# Example CUDA 11.7, Pytorch 2.0.1 (replace with your required pytorch version if different) -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -``` - -### Install dependencies -```bash -# CPU only (reccomended to attempt first) -pip install .[cpu,dev] - -# CUDA install -pip install .[cuda,dev] - -# Copy and configure environment variables -cp default.env .env -``` - -### Test install -```bash -# Run standard tests -pytest -s dimos/ - -# Test modules functionality -pytest -s -m module dimos/ - -# Test LCM communication -pytest -s -m lcm dimos/ -``` - -# Unitree Go2 Quickstart - -To quickly test the modules system, you can run the Unitree Go2 multiprocess example directly: - -```bash -# Make sure you have the required environment variables set -export ROBOT_IP= - -# Run the multiprocess Unitree Go2 example -python dimos/robot/unitree_webrtc/multiprocess/unitree_go2.py -``` diff --git a/docs/old/modules_CN.md b/docs/old/modules_CN.md deleted file mode 100644 index 89e16c7112..0000000000 --- a/docs/old/modules_CN.md +++ /dev/null @@ -1,188 +0,0 @@ -# Dimensional 模块系统 - -DimOS 模块系统使用 Dask 进行计算分布和 LCM（轻量级通信和编组）进行高性能进程间通信，实现分布式、多进程的机器人应用。 - -## 核心概念 - -### 1. 模块定义 -模块是继承自 `dimos.core.Module` 的 Python 类，定义输入、输出和 RPC 方法： - -```python -from dimos.core import Module, In, Out, rpc -from dimos.msgs.geometry_msgs import Vector3 - -class MyModule(Module): # ROS Node - # 将输入/输出声明为初始化为 None 的类属性 - data_in: In[Vector3] = None # ROS Subscriber - data_out: Out[Vector3] = None # ROS Publisher - - def __init__(): - # 调用父类 Module 初始化 - super().__init__() - - @rpc - def remote_method(self, param): - """使用 @rpc 装饰的方法可以远程调用""" - return param * 2 -``` - -### 2. 模块部署 -使用 `dimos.deploy()` 方法在 Dask 工作进程中部署模块： - -```python -from dimos import core - -# 启动具有 N 个工作进程的 Dask 集群 -dimos = core.start(4) - -# 部署模块时可以传递初始化参数 -# 在这种情况下，param1 和 param2 被传递到模块初始化中 -module = dimos.deploy(Module, param1="value1", param2=123) -``` - -### 3. 流连接 -模块通过使用 LCM 传输的响应式流进行通信： - -```python -# 为输出配置 LCM 传输 -module1.data_out.transport = core.LCMTransport("/topic_name", MessageType) - -# 将模块输入连接到输出 -module2.data_in.connect(module1.data_out) - -# 访问底层的 Observable 流 -stream = module1.data_out.observable() -stream.subscribe(lambda msg: print(f"接收到: {msg}")) -``` - -### 4. 模块生命周期 -```python -# 启动模块以开始处理 -module.start() # 如果定义了 @rpc start() 方法，则调用它 - -# 检查模块 I/O 配置 -print(module.io().result()) # 显示输入、输出和 RPC 方法 - -# 优雅关闭 -dimos.shutdown() -``` - -## 实际示例：机器人控制系统 - -```python -# 连接模块封装机器人硬件/仿真 -connection = dimos.deploy(ConnectionModule, ip=robot_ip) -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) -connection.video.transport = core.LCMTransport("/video", Image) - -# 感知模块处理传感器数据 -perception = dimos.deploy(PersonTrackingStream, camera_intrinsics=[...]) -perception.video.connect(connection.video) -perception.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# 开始处理 -connection.start() -perception.start() - -# 通过 RPC 启用跟踪 -perception.enable_tracking() - -# 获取最新的跟踪数据 -data = perception.get_tracking_data() -``` - -## LCM 传输配置 - -```python -# 用于简单类型（如激光雷达）的标准 LCM 传输 -connection.lidar.transport = core.LCMTransport("/lidar", LidarMessage) - -# 用于复杂 Python 对象/字典的基于 pickle 的传输 -connection.tracking_data.transport = core.pLCMTransport("/person_tracking") - -# 自动配置 LCM 系统缓冲区（在容器中必需） -from dimos.protocol import pubsub -pubsub.lcm.autoconf() -``` - -这种架构使得能够将复杂的机器人系统构建为可组合的分布式模块，这些模块通过流和 RPC 高效通信，从单机扩展到集群。 - -# Dimensional 安装指南 -## Python 安装（Ubuntu 22.04） - -```bash -sudo apt install python3-venv - -# 克隆仓库（dev 分支，无子模块） -git clone -b dev https://github.com/dimensionalOS/dimos.git -cd dimos - -# 创建并激活虚拟环境 -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# 如果尚未安装，请安装 torch 和 torchvision -# 示例 CUDA 11.7，Pytorch 2.0.1（如果需要不同的 pytorch 版本，请替换） -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -``` - -### 安装依赖 -```bash -# 仅 CPU（建议首先尝试） -pip install .[cpu,dev] - -# CUDA 安装 -pip install .[cuda,dev] - -# 复制并配置环境变量 -cp default.env .env -``` - -### 测试安装 -```bash -# 运行标准测试 -pytest -s dimos/ - -# 测试模块功能 -pytest -s -m module dimos/ - -# 测试 LCM 通信 -pytest -s -m lcm dimos/ -``` - -# Unitree Go2 快速开始 - -要快速测试模块系统，您可以直接运行 Unitree Go2 多进程示例： - -```bash -# 确保设置了所需的环境变量 -export ROBOT_IP= - -# 运行多进程 Unitree Go2 示例 -python dimos/robot/unitree_webrtc/multiprocess/unitree_go2.py -``` - -## 模块系统的高级特性 - -### 分布式计算 -DimOS 模块系统建立在 Dask 之上，提供了强大的分布式计算能力： - -- **自动负载均衡**：模块自动分布在可用的工作进程中 -- **容错性**：如果工作进程失败，模块可以在其他工作进程上重新启动 -- **可扩展性**：从单机到集群的无缝扩展 - -### 响应式编程模型 -使用 RxPY 实现的响应式流提供了： - -- **异步处理**：非阻塞的数据流处理 -- **背压处理**：自动管理快速生产者和慢速消费者 -- **操作符链**：使用 map、filter、merge 等操作符进行流转换 - -### 性能优化 -LCM 传输针对机器人应用进行了优化： - -- **零拷贝**：大型消息的高效内存使用 -- **低延迟**：微秒级的消息传递 -- **多播支持**：一对多的高效通信 diff --git a/docs/old/ros_navigation.md b/docs/old/ros_navigation.md deleted file mode 100644 index 4a74500b2f..0000000000 --- a/docs/old/ros_navigation.md +++ /dev/null @@ -1,284 +0,0 @@ -# Autonomy Stack API Documentation - -## Prerequisites - -- Ubuntu 24.04 -- [ROS 2 Jazzy Installation](https://docs.ros.org/en/jazzy/Installation.html) - -Add the following line to your `~/.bashrc` to source the ROS 2 Jazzy setup script automatically: - -``` echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc``` - -## MID360 Ethernet Configuration (skip for sim) - -### Step 1: Configure Network Interface - -1. Open Network Settings in Ubuntu -2. Find your Ethernet connection to the MID360 -3. Click the gear icon to edit settings -4. Go to IPv4 tab -5. Change Method from "Automatic (DHCP)" to "Manual" -6. Add the following settings: - - **Address**: 192.168.1.5 - - **Netmask**: 255.255.255.0 - - **Gateway**: 192.168.1.1 -7. Click "Apply" - -### Step 2: Configure MID360 IP in JSON - -1. Find your MID360 serial number (on sticker under QR code) -2. Note the last 2 digits (e.g., if serial ends in 89, use 189) -3. Edit the configuration file: - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform -nano src/utilities/livox_ros_driver2/config/MID360_config.json -``` - -4. Update line 28 with your IP (192.168.1.1xx where xx = last 2 digits): - -```json -"ip" : "192.168.1.1xx", -``` - -5. Save and exit - -### Step 3: Verify Connection - -```bash -ping 192.168.1.1xx # Replace xx with your last 2 digits -``` - -## Robot Configuration - -### Setting Robot Type - -The system supports different robot configurations. Set the `ROBOT_CONFIG_PATH` environment variable to specify which robot configuration to use: - -```bash -# For Unitree G1 (default if not set) -export ROBOT_CONFIG_PATH="unitree/unitree_g1" - -# Add to ~/.bashrc to make permanent -echo 'export ROBOT_CONFIG_PATH="unitree/unitree_g1"' >> ~/.bashrc -``` - -Available robot configurations: -- `unitree/unitree_g1` - Unitree G1 robot (default) -- Add your custom robot configs in `src/base_autonomy/local_planner/config/` - -## Build the system - -You must do this every you make a code change, this is not Python - -```colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release``` - -## System Launch - -### Simulation Mode - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform - -# Base autonomy only -./system_simulation.sh - -# With route planner -./system_simulation_with_route_planner.sh - -# With exploration planner -./system_simulation_with_exploration_planner.sh -``` - -### Real Robot Mode - -```bash -cd ~/autonomy_stack_mecanum_wheel_platform - -# Base autonomy only -./system_real_robot.sh - -# With route planner -./system_real_robot_with_route_planner.sh - -# With exploration planner -./system_real_robot_with_exploration_planner.sh -``` - -## Quick Troubleshooting - -- **Cannot ping MID360**: Check Ethernet cable and network settings -- **SLAM drift**: Press clear-terrain-map button on joystick controller -- **Joystick not recognized**: Unplug and replug USB dongle - - -## ROS Topics - -### Input Topics (Commands) - -| Topic | Type | Description | -|-------|------|-------------| -| `/way_point` | `geometry_msgs/PointStamped` | Send navigation goal (position only) | -| `/goal_pose` | `geometry_msgs/PoseStamped` | Send goal with orientation | -| `/cancel_goal` | `std_msgs/Bool` | Cancel current goal (data: true) | -| `/joy` | `sensor_msgs/Joy` | Joystick input | -| `/stop` | `std_msgs/Int8` | Soft Stop (2=stop all commmand, 0 = release) | -| `/navigation_boundary` | `geometry_msgs/PolygonStamped` | Set navigation boundaries | -| `/added_obstacles` | `sensor_msgs/PointCloud2` | Virtual obstacles | - -### Output Topics (Status) - -| Topic | Type | Description | -|-------|------|-------------| -| `/state_estimation` | `nav_msgs/Odometry` | Robot pose from SLAM | -| `/registered_scan` | `sensor_msgs/PointCloud2` | Aligned lidar point cloud | -| `/terrain_map` | `sensor_msgs/PointCloud2` | Local terrain map | -| `/terrain_map_ext` | `sensor_msgs/PointCloud2` | Extended terrain map | -| `/path` | `nav_msgs/Path` | Local path being followed | -| `/cmd_vel` | `geometry_msgs/Twist` | Velocity commands to motors | -| `/goal_reached` | `std_msgs/Bool` | True when goal reached, false when cancelled/new goal | - -### Map Topics - -| Topic | Type | Description | -|-------|------|-------------| -| `/overall_map` | `sensor_msgs/PointCloud2` | Global map (only in sim)| -| `/registered_scan` | `sensor_msgs/PointCloud2` | Current scan in map frame | -| `/terrain_map` | `sensor_msgs/PointCloud2` | Local obstacle map | - -## Usage Examples - -### Send Goal -```bash -ros2 topic pub /way_point geometry_msgs/msg/PointStamped "{ - header: {frame_id: 'map'}, - point: {x: 5.0, y: 3.0, z: 0.0} -}" --once -``` - -### Cancel Goal -```bash -ros2 topic pub /cancel_goal std_msgs/msg/Bool "data: true" --once -``` - -### Monitor Robot State -```bash -ros2 topic echo /state_estimation -``` - -## Configuration Parameters - -### Vehicle Parameters (`localPlanner`) - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `vehicleLength` | 0.5 | Robot length (m) | -| `vehicleWidth` | 0.5 | Robot width (m) | -| `maxSpeed` | 0.875 | Maximum speed (m/s) | -| `autonomySpeed` | 0.875 | Autonomous mode speed (m/s) | - -### Goal Tolerance Parameters - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `goalReachedThreshold` | 0.3-0.5 | Distance to consider goal reached (m) | -| `goalClearRange` | 0.35-0.6 | Extra clearance around goal (m) | -| `goalBehindRange` | 0.35-0.8 | Stop pursuing if goal behind within this distance (m) | -| `omniDirGoalThre` | 1.0 | Distance for omnidirectional approach (m) | - -### Obstacle Avoidance - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `obstacleHeightThre` | 0.1-0.2 | Height threshold for obstacles (m) | -| `adjacentRange` | 3.5 | Sensor range for planning (m) | -| `minRelZ` | -0.4 | Minimum relative height to consider (m) | -| `maxRelZ` | 0.3 | Maximum relative height to consider (m) | - -### Path Planning - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `pathScale` | 0.875 | Path resolution scale | -| `minPathScale` | 0.675 | Minimum path scale when blocked | -| `minPathRange` | 0.8 | Minimum planning range (m) | -| `dirThre` | 90.0 | Direction threshold (degrees) | - -### Control Parameters (`pathFollower`) - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `lookAheadDis` | 0.5 | Look-ahead distance (m) | -| `maxAccel` | 2.0 | Maximum acceleration (m/s²) | -| `slowDwnDisThre` | 0.875 | Slow down distance threshold (m) | - -### SLAM Blind Zones (`feature_extraction_node`) - -| Parameter | Mecanum | Description | -|-----------|---------|-------------| -| `blindFront` | 0.1 | Front blind zone (m) | -| `blindBack` | -0.2 | Back blind zone (m) | -| `blindLeft` | 0.1 | Left blind zone (m) | -| `blindRight` | -0.1 | Right blind zone (m) | -| `blindDiskRadius` | 0.4 | Cylindrical blind zone radius (m) | - -## Operating Modes - -### Mode Control -- **Joystick L2**: Hold for autonomy mode -- **Joystick R2**: Hold to disable obstacle checking - -### Speed Control -The robot automatically adjusts speed based on: -1. Obstacle proximity -2. Path complexity -3. Goal distance - -## Tuning Guide - -### For Tighter Navigation -- Decrease `goalReachedThreshold` (e.g., 0.2) -- Decrease `goalClearRange` (e.g., 0.3) -- Decrease `vehicleLength/Width` slightly - -### For Smoother Navigation -- Increase `goalReachedThreshold` (e.g., 0.5) -- Increase `lookAheadDis` (e.g., 0.7) -- Decrease `maxAccel` (e.g., 1.5) - -### For Aggressive Obstacle Avoidance -- Increase `obstacleHeightThre` (e.g., 0.15) -- Increase `adjacentRange` (e.g., 4.0) -- Increase blind zone parameters - -## Common Issues - -### Robot Oscillates at Goal -- Increase `goalReachedThreshold` -- Increase `goalBehindRange` - -### Robot Stops Too Far from Goal -- Decrease `goalReachedThreshold` -- Decrease `goalClearRange` - -### Robot Hits Low Obstacles -- Decrease `obstacleHeightThre` -- Adjust `minRelZ` to include lower points - -## SLAM Configuration - -### Localization Mode -Set in `livox_mid360.yaml`: -```yaml -local_mode: true -init_x: 0.0 -init_y: 0.0 -init_yaw: 0.0 -``` - -### Mapping Performance -```yaml -mapping_line_resolution: 0.1 # Decrease for higher quality -mapping_plane_resolution: 0.2 # Decrease for higher quality -max_iterations: 5 # Increase for better accuracy -``` diff --git a/docs/old/running_without_devcontainer.md b/docs/old/running_without_devcontainer.md deleted file mode 100644 index d06785e359..0000000000 --- a/docs/old/running_without_devcontainer.md +++ /dev/null @@ -1,21 +0,0 @@ -install nix, - -https://nixos.wiki/wiki/Nix_Installation_Guide -```sh -sudo install -d -m755 -o $(id -u) -g $(id -g) /nix -curl -L https://nixos.org/nix/install | sh -``` - -install direnv -https://direnv.net/ -```sh -apt-get install direnv -echo 'eval "$(direnv hook bash)"' >> ~/.bashrc -``` - -allow direnv in dimos will take a bit to pull the packages, -from that point on your env is standardized -```sh -cd dimos -direnv allow -``` diff --git a/docs/old/testing_stream_reply.md b/docs/old/testing_stream_reply.md deleted file mode 100644 index e3189bb5e8..0000000000 --- a/docs/old/testing_stream_reply.md +++ /dev/null @@ -1,174 +0,0 @@ -# Sensor Replay & Storage Toolkit - -A lightweight framework for **recording, storing, and replaying binary data streams for automated tests**. It keeps your repository small (data lives in Git LFS) while giving you Python‑first ergonomics for working with RxPY streams, point‑clouds, videos, command logs—anything you can pickle. - ---- - -## 1 At a Glance - -| Need | One liner | -| ------------------------------ | ------------------------------------------------------------- | -| **Iterate over every message** | `SensorReplay("raw_odometry_rotate_walk").iterate(print)` | -| **RxPY stream for piping** | `SensorReplay("raw_odometry_rotate_walk").stream().pipe(...)` | -| **Throttle replay rate** | `SensorReplay("raw_odometry_rotate_walk").stream(rate_hz=10)` | -| **Raw path to a blob/dir** | `path = testData("raw_odometry_rotate_walk")` | -| **Store a new stream** | see [`SensorStorage`](#5-storing-new-streams) | - -> If the requested blob is missing locally, it is transparently downloaded from Git LFS, extracted to `tests/data//`, and cached for subsequent runs. - ---- - -## 2 Goals - -* **Zero setup for CI & collaborators** – data is fetched on demand. -* **No repo bloat** – binaries live in Git LFS; the working tree stays trim. -* **Symmetric API** – `SensorReplay` ↔︎ `SensorStorage`; same name, different direction. -* **Format agnostic** – replay *anything* you can pickle (protobuf, numpy, JPEG, …). -* **Data type agnostic** – with testData("raw_odometry_rotate_walk") you get a Path object back, can be a raw video file, whole codebase, ML model etc - - ---- - -## 3 Replaying Data - -### 3.1 Iterating Messages - -```python -from sensor_tools import SensorReplay - -# Print every stored Odometry message -SensorReplay(name="raw_odometry_rotate_walk").iterate(print) -``` - -### 3.2 RxPY Streaming - -```python -from rx import operators as ops -from operator import sub, add -from dimos.utils.testing import SensorReplay, SensorStorage -from dimos.robot.unitree_webrtc.type.odometry import Odometry - -# Compute total yaw rotation (radians) - -total_rad = ( - SensorReplay("raw_odometry_rotate_walk", autocast=Odometry.from_msg) - .stream() - .pipe( - ops.map(lambda odom: odom.rot.z), - ops.pairwise(), # [1,2,3,4] -> [[1,2], [2,3], [3,4]] - ops.starmap(sub), # [sub(1,2), sub(2,3), sub(3,4)] - ops.reduce(add), - ) - .run() -) - -assert total_rad == pytest.approx(4.05, abs=0.01) -``` - -### 3.3 Lidar Mapping Example (200MB blob) - -```python -from dimos.utils.testing import SensorReplay, SensorStorage -from dimos.robot.unitree_webrtc.type.map import Map - -lidar_stream = SensorReplay("office_lidar", autocast=LidarMessage.from_msg) -map_ = Map(voxel_size=0.5) - -# Blocks until the stream is consumed -map_.consume(lidar_stream.stream()).run() - -assert map_.costmap.grid.shape == (404, 276) -``` - ---- - -## 4 Low Level Access - -If you want complete control, call **`testData(name)`** to get a `Path` to the extracted file or directory — no pickling assumptions: - -```python -absolute_path: Path = testData("some_name") -``` - -Do whatever you like: open a video file, load a model checkpoint, etc. - ---- - -## 5 Storing New Streams - -1. **Write a test marked `@pytest.mark.tool`** so CI skips it by default. -2. Use `SensorStorage` to persist the stream into `tests/data//*.pickle`. - -```python -@pytest.mark.tool -def test_store_odometry_stream(): - load_dotenv() - - robot = UnitreeGo2(ip=os.getenv("ROBOT_IP"), mode="ai") - robot.standup() - - storage = SensorStorage("raw_odometry_rotate_walk2") - storage.save_stream(robot.raw_odom_stream()) # ← records until interrupted - - try: - while True: - time.sleep(0.1) - except KeyboardInterrupt: - robot.liedown() -``` - -### 5.1 Behind the Scenes - -* Any new file/dir under `tests/data/` is treated as a **data blob**. -* `./bin/lfs_push` compresses it into `tests/data/.lfs/.tar.gz` *and* uploads it to Git LFS. -* Only the `.lfs/` archive is committed; raw binaries remain `.gitignored`. - ---- - -## 6 Storing Arbitrary Binary Data - -Just copy to `tests/data/whatever` -* `./bin/lfs_push` compresses it into `tests/data/.lfs/.tar.gz` *and* uploads it to Git LFS. - ---- - -## 7 Developer Workflow Checklist - -1. **Drop new data** into `tests/data/`. -2. Run your new tests that use SensorReplay or testData calls, make sure all works -3. Run `./bin/lfs_push` (or let the pre commit hook nag you). -4. Commit the resulting `tests/data/.lfs/.tar.gz`. -5. Optional - you can delete `tests/data/your_new_stuff` and re-run the test to ensure it gets downloaded from LFS correclty -6. Push/PR - -### 7.1 Pre commit Setup (optional but recommended) - -```sh -sudo apt install pre-commit -pre-commit install # inside repo root -``` - -Now each commit checks formatting, linting, *and* whether you forgot to push new blobs: - -``` -$ echo test > tests/data/foo.txt -$ git add tests/data/foo.txt && git commit -m "demo" -LFS data ......................................................... Failed -✗ New test data detected at /tests/data: - foo.txt -Either delete or run ./bin/lfs_push -``` - ---- - -## 8 Future Work - -- A replay rate that mirrors the **original message timestamps** can be implemented downstream (e.g., an RxPY operator) -- Likely this same system should be used for production binary data delivery as well (Models etc) - ---- - -## 9 Existing Examples - -* `dimos/robot/unitree_webrtc/type/test_odometry.py` -* `dimos/robot/unitree_webrtc/type/test_map.py` diff --git a/docs/package_usage.md b/docs/package_usage.md deleted file mode 100644 index 24584a2e79..0000000000 --- a/docs/package_usage.md +++ /dev/null @@ -1,62 +0,0 @@ -# Package Usage - -## With `uv` - -Init your repo if not already done: - -```bash -uv init -``` - -Install: - -```bash -uv add dimos[dev,cpu,sim] -``` - -Test the Unitree Go2 robot in the simulator: - -```bash -uv run dimos-robot --simulation run unitree-g1 -``` - -Run your actual robot: - -```bash -uv run dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` - -### Without installing - -With `uv` you can run tools without having to explicitly install: - -```bash -uvx --from dimos dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` - -## With `pip` - -Create an environment if not already done: - -```bash -python -m venv .venv -. .venv/bin/activate -``` - -Install: - -```bash -pip install dimos[dev,cpu,sim] -``` - -Test the Unitree Go2 robot in the simulator: - -```bash -dimos-robot --simulation run unitree-g1 -``` - -Run your actual robot: - -```bash -dimos-robot --robot-ip=192.168.X.XXX run unitree-g1 -``` From 889eba85741f450e7855e415d6e13750eaa9c6ef Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 03:21:37 -0800 Subject: [PATCH 040/136] minimal edit --- README.md | 529 ++++++++---------------------------- docs/concepts/blueprints.md | 319 ++++++++++++++++++++++ docs/development/README.md | 282 +++++++++++++++++++ 3 files changed, 708 insertions(+), 422 deletions(-) create mode 100644 docs/concepts/blueprints.md create mode 100644 docs/development/README.md diff --git a/README.md b/README.md index 9a74d63aa7..4dbfc8cac3 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -![Screenshot 2025-02-18 at 16-31-22 DimOS Terminal](/assets/dimos_terminal.png) + + # The Dimensional Framework *The universal framework for AI-native generalist robotics* ## What is Dimensional? -Dimensional is an open-source framework for building agentive generalist robots. DimOS allows off-the-shelf Agents to call tools/functions and read sensor/state data directly from ROS. +#### Warning: This is a pre-release version -The framework enables neurosymbolic orchestration of Agents as generalized spatial reasoners/planners and Robot state/action primitives as functions. +Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. -The result: cross-embodied *"Dimensional Applications"* exceptional at generalization and robust at symbolic action execution. +# How do I get started? -## DIMOS x Unitree Go2 (OUT OF DATE) +## Installation -We are shipping a first look at the DIMOS x Unitree Go2 integration, allowing for off-the-shelf Agents() to "call" Unitree ROS2 Nodes and WebRTC action primitives, including: +#### Details / Requirements -- Navigation control primitives (move, reverse, spinLeft, spinRight, etc.) -- WebRTC control primitives (FrontPounce, FrontFlip, FrontJump, etc.) -- Camera feeds (image_raw, compressed_image, etc.) -- IMU data -- State information -- Lidar / PointCloud primitives -- Any other generic Unitree ROS2 topics - -### Features - -- **DimOS Agents** - - Agent() classes with planning, spatial reasoning, and Robot.Skill() function calling abilities. - - Integrate with any off-the-shelf hosted or local model: OpenAIAgent, ClaudeAgent, GeminiAgent 🚧, DeepSeekAgent 🚧, HuggingFaceRemoteAgent, HuggingFaceLocalAgent, etc. - - Modular agent architecture for easy extensibility and chaining of Agent output --> Subagents input. - - Agent spatial / language memory for location grounded reasoning and recall. - -- **DimOS Infrastructure** - - A reactive data streaming architecture using RxPY to manage real-time video (or other sensor input), outbound commands, and inbound robot state between the DimOS interface, Agents, and ROS2. - - Robot Command Queue to handle complex multi-step actions to Robot. - - Simulation bindings (Genesis, Isaacsim, etc.) to test your agentive application before deploying to a physical robot. - -- **DimOS Interface / Development Tools** - - Local development interface to control your robot, orchestrate agents, visualize camera/lidar streams, and debug your dimensional agentive application. - -## MacOS Installation +- Linux, tested on Ubuntu 22.04, 24.04 +- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* + - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` ```sh -# Install Nix -curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install - -# clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git - -# setup the environment (follow the prompts after nix develop) -cd dimos -nix develop - -# You should be able to follow the instructions below as well for a more manual installation -``` - ---- -## Python Installation -Tested on Ubuntu 22.04/24.04 - -```bash -sudo apt install python3-venv - -# Clone the repository -git clone --branch dev --single-branch https://github.com/dimensionalOS/dimos.git -cd dimos - -# Create and activate virtual environment -python3 -m venv venv -source venv/bin/activate - -sudo apt install portaudio19-dev python3-pyaudio - -# Install LFS -sudo apt install git-lfs -git lfs install - -# Install torch and torchvision if not already installed -# Example CUDA 11.7, Pytorch 2.0.1 (replace with your required pytorch version if different) -pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 +sudo apt-get update +sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" + +# +# NOTE!!! the first time, you're going to have an empty/black rerun window for a while +# +# the command needs to download the replay file (2.4gb), which takes a bit + +# OPTION 1: use without installing dimos +uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 + +# OPTION 2: install dimos in a virtualenv +uv venv && . .venv/bin/activate +uv pip install 'dimos[base,unitree]' +dimos --replay run unitree-go2 ``` -#### Install dependencies -```bash -# CPU only (reccomended to attempt first) -pip install -e '.[cpu,dev]' - -# CUDA install -pip install -e '.[cuda,dev]' + -# Copy and configure environment variables -cp default.env .env -``` +#### Get it working on a real physical robot! -#### Test the install -```bash -pytest -s dimos/ +```sh +export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +dimos run unitree-go2 ``` -#### Test Dimensional with a replay UnitreeGo2 stream (no robot required) -```bash -dimos --replay run unitree-go2 -``` +#### Get it working in an interactive simulation! -#### Test Dimensional with a simulated UnitreeGo2 in MuJoCo (no robot required) ```bash -pip install -e '.[sim]' export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors +# ignore the warp warnings dimos --simulation run unitree-go2 +# open http://localhost:7779/command-center in your browser to control the robot movement ``` -#### Test Dimensional with a real UnitreeGo2 over WebRTC -```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2 -``` +#### Have it controlled by AI! -#### Test Dimensional with a real UnitreeGo2 running Agents -*OpenAI / Alibaba keys required* ```bash -export ROBOT_IP=192.168.X.XXX # Add the robot IP address -dimos run unitree-go2-agentic -``` ---- - -### Agent API keys - -Full functionality will require API keys for the following: - -Requirements: -- OpenAI API key (required for all LLMAgents due to OpenAIEmbeddings) -- Claude API key (required for ClaudeAgent) -- Alibaba API key (required for Navigation skills) - -These keys can be added to your .env file or exported as environment variables. -``` export OPENAI_API_KEY= -export CLAUDE_API_KEY= -export ALIBABA_API_KEY= -``` - -### ROS2 Unitree Go2 SDK Installation - -#### System Requirements -- Ubuntu 22.04 -- ROS2 Distros: Iron, Humble, Rolling - -See [Unitree Go2 ROS2 SDK](https://github.com/dimensionalOS/go2_ros2_sdk) for additional installation instructions. - -```bash -mkdir -p ros2_ws -cd ros2_ws -git clone --recurse-submodules https://github.com/dimensionalOS/go2_ros2_sdk.git src -sudo apt install ros-$ROS_DISTRO-image-tools -sudo apt install ros-$ROS_DISTRO-vision-msgs - -sudo apt install python3-pip clang portaudio19-dev -cd src -pip install -r requirements.txt -cd .. - -# Ensure clean python install before running -source /opt/ros/$ROS_DISTRO/setup.bash -rosdep install --from-paths src --ignore-src -r -y -colcon build -``` - -### Run the test application - -#### ROS2 Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -source /opt/ros/$ROS_DISTRO/setup.bash - -export ROBOT_IP="robot_ip" #for muliple robots, just split by , -export CONN_TYPE="webrtc" -ros2 launch go2_robot_sdk robot.launch.py - -``` - -#### Python Terminal: -```bash -# Change path to your Go2 ROS2 SDK installation -source /ros2_ws/install/setup.bash -python tests/run.py -``` - -#### DimOS Interface: -```bash -cd dimos/web/dimos_interface -yarn install -yarn dev # you may need to run sudo if previously built via Docker -``` - -### Project Structure (OUT OF DATE) - -``` -. -├── dimos/ -│ ├── agents/ # Agent implementations -│ │ └── memory/ # Memory systems for agents, including semantic memory -│ ├── environment/ # Environment context and sensing -│ ├── hardware/ # Hardware abstraction and interfaces -│ ├── models/ # ML model definitions and implementations -│ │ ├── Detic/ # Detic object detection model -│ │ ├── depth/ # Depth estimation models -│ │ ├── segmentation/ # Image segmentation models -│ ├── perception/ # Computer vision and sensing -│ │ ├── detection2d/ # 2D object detection -│ │ └── segmentation/ # Image segmentation pipelines -│ ├── robot/ # Robot control and hardware interface -│ │ ├── global_planner/ # Path planning at global scale -│ │ ├── local_planner/ # Local navigation planning -│ │ └── unitree/ # Unitree Go2 specific implementations -│ ├── simulation/ # Robot simulation environments -│ │ ├── genesis/ # Genesis simulation integration -│ │ └── isaac/ # NVIDIA Isaac Sim integration -│ ├── skills/ # Task-specific robot capabilities -│ │ └── rest/ # REST API based skills -│ ├── stream/ # WebRTC and data streaming -│ │ ├── audio/ # Audio streaming components -│ │ └── video_providers/ # Video streaming components -│ ├── types/ # Type definitions and interfaces -│ ├── utils/ # Utility functions and helpers -│ └── web/ # DimOS development interface and API -│ ├── dimos_interface/ # DimOS web interface -│ └── websocket_vis/ # Websocket visualizations -├── tests/ # Test files -│ ├── genesissim/ # Genesis simulator tests -│ └── isaacsim/ # Isaac Sim tests -└── docker/ # Docker configuration files - ├── agent/ # Agent service containers - ├── interface/ # Interface containers - ├── simulation/ # Simulation environment containers - └── unitree/ # Unitree robot specific containers +dimos run unitree-go2-agentic +# open the following in a different terminal tab to tell it where to go +# Warning!: make sure to watch the bot, this is a pre-release it will run into stuff +# and get tangled/trapped +# ex: tell it to explore the room, then tell it to go to where it can see a door +humancli ``` -## Building +# How do I use it as a library? -### Simple DimOS Application (OUT OF DATE) -```python -from dimos.robot.unitree.unitree_go2 import UnitreeGo2 -from dimos.robot.unitree.unitree_skills import MyUnitreeSkills -from dimos.robot.unitree.unitree_ros_control import UnitreeROSControl -from dimos.agents_deprecated.agent import OpenAIAgent +Simple camera activation (save this as a python file and run it): -# Initialize robot -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) +```py +from dimos.core.blueprints import autoconnect +from dimos.hardware.camera.module import CameraModule -# Initialize agent -agent = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_video_stream=robot.get_ros_video_stream(), - skills=robot.get_skills(), - system_query="Jump when you see a human! Front flip when you see a dog!", - model_name="gpt-4o" - ) - -while True: # keep process running - time.sleep(1) +if __name__ == "__main__": + autoconnect( + CameraModule.blueprint() + ).build().loop() + print("Webcam pipeline running. Press Ctrl+C to stop.") ``` +Write your own custom module: -### DimOS Application with Agent chaining (OUT OF DATE) - -Let's build a simple DimOS application with Agent chaining. We define a ```planner``` as a ```PlanningAgent``` that takes in user input to devise a complex multi-step plan. This plan is passed step-by-step to an ```executor``` agent that can queue ```AbstractRobotSkill``` commands to the ```ROSCommandQueue```. +```py +from dimos.core.blueprints import autoconnect +from dimos.core import In, Module, pSHMTransport +from dimos.core.core import rpc +from dimos.hardware.camera.module import CameraModule +from dimos.msgs.sensor_msgs import Image -Our reactive Pub/Sub data streaming architecture allows for chaining of ```Agent_0 --> Agent_1 --> ... --> Agent_n``` via the ```input_query_stream``` parameter in each which takes an ```Observable``` input from the previous Agent in the chain. +from reactivex.disposable import Disposable -**Via this method you can chain together any number of Agents() to create complex dimensional applications.** +class Listener(Module): + # the CameraModule has an Out[Image] named "color_image" + # this module will only receive those messages if + # the names ("color_image") match, otherwise autoconnect + # will not be able to connect one module to another + color_image: In[Image] = None + grayscale_image: Out[Image] = None -```python + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.count = 0 -web_interface = RobotWebInterface(port=5555) + @rpc + def start(self) -> None: + super().start() + def callback_func(img: Image) -> None: + self.count += 1 + print(f"got frame {self.count}") + print(f"img.data.shape: {img.data.shape}") + self.grayscale_image.publish(img.to_grayscale()) -robot = UnitreeGo2(ip=robot_ip, - ros_control=UnitreeROSControl(), - skills=MyUnitreeSkills()) + unsubscribe_func = self.color_image.subscribe(callback_func) + # disposables will be called when the module is stopped + self._disposables.add(Disposable( + unsubscribe_func + )) -# Initialize master planning agent -planner = PlanningAgent( - dev_name="UnitreePlanningAgent", - input_query_stream=web_interface.query_stream, # Takes user input from dimOS interface - skills=robot.get_skills(), - model_name="gpt-4o", - ) + @rpc + def stop(self) -> None: + super().stop() -# Initialize execution agent -executor = OpenAIAgent( - dev_name="UnitreeExecutionAgent", - input_query_stream=planner.get_response_observable(), # Takes planner output as input - skills=robot.get_skills(), - model_name="gpt-4o", - system_query=""" - You are a robot execution agent that can execute tasks on a virtual - robot. ONLY OUTPUT THE SKILLS TO EXECUTE. - """ - ) - -while True: # keep process running - time.sleep(1) +if __name__ == "__main__": + autoconnect( + Listener.blueprint(), + CameraModule.blueprint(), + ).build().loop() ``` -### Calling Action Primitives (OUT OF DATE) - -Call action primitives directly from ```Robot()``` for prototyping and testing. - -```python -robot = UnitreeGo2(ip=robot_ip,) - -# Call a Unitree WebRTC action primitive -robot.webrtc_req(api_id=1016) # "Hello" command - -# Call a ROS2 action primitive -robot.move(distance=1.0, speed=0.5) -``` - -### Creating Custom Skills (non-unitree specific) - -#### Create basic custom skills by inheriting from ```AbstractRobotSkill``` and implementing the ```__call__``` method. - -```python -class Move(AbstractRobotSkill): - distance: float = Field(...,description="Distance to reverse in meters") - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - return self._robot.move(distance=self.distance) -``` - -#### Chain together skills to create recursive skill trees - -```python -class JumpAndFlip(AbstractRobotSkill): - def __init__(self, robot: Optional[Robot] = None, **data): - super().__init__(robot=robot, **data) - def __call__(self): - super().__call__() - jump = Jump(robot=self._robot) - flip = Flip(robot=self._robot) - return (jump() and flip()) -``` - -### Integrating Skills with Agents: Single Skills and Skill Libraries - -DimOS agents, such as `OpenAIAgent`, can be endowed with capabilities through two primary mechanisms: by providing them with individual skill classes or with comprehensive `SkillLibrary` instances. This design offers flexibility in how robot functionalities are defined and managed within your agent-based applications. - -**Agent's `skills` Parameter** - -The `skills` parameter in an agent's constructor is key to this integration: - -1. **A Single Skill Class**: This approach is suitable for skills that are relatively self-contained or have straightforward initialization requirements. - * You pass the skill *class itself* (e.g., `GreeterSkill`) directly to the agent's `skills` parameter. - * The agent then takes on the responsibility of instantiating this skill when it's invoked. This typically involves the agent providing necessary context to the skill's constructor (`__init__`), such as a `Robot` instance (or any other private instance variable) if the skill requires it. - -2. **A `SkillLibrary` Instance**: This is the preferred method for managing a collection of skills, especially when skills have dependencies, require specific configurations, or need to share parameters. - * You first define your custom skill library by inheriting from `SkillLibrary`. Then, you create and configure an *instance* of this library (e.g., `my_lib = EntertainmentSkills(robot=robot_instance)`). - * This pre-configured `SkillLibrary` instance is then passed to the agent's `skills` parameter. The library itself manages the lifecycle and provision of its contained skills. +### Note: Many More Examples in the [Examples Folder](./examples) -**Examples:** -#### 1. Using a Single Skill Class with an Agent +# How does DimOS work conceptually? -First, define your skill. For instance, a `GreeterSkill` that can deliver a configurable greeting: +There are several tools: +- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. +- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other +- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Skills](/docs/concepts/blueprints.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools -```python -class GreeterSkill(AbstractSkill): - """Greats the user with a friendly message.""" # Gives the agent better context for understanding (the more detailed the better). +# Contributing / Building From Source - greeting: str = Field(..., description="The greating message to display.") # The field needed for the calling of the function. Your agent will also pull from the description here to gain better context. - - def __init__(self, greeting_message: Optional[str] = None, **data): - super().__init__(**data) - if greeting_message: - self.greeting = greeting_message - # Any additional skill-specific initialization can go here - - def __call__(self): - super().__call__() # Call parent's method if it contains base logic - # Implement the logic for the skill - print(self.greeting) - return f"Greeting delivered: '{self.greeting}'" -``` - -Next, register this skill *class* directly with your agent. The agent can then instantiate it, potentially with specific configurations if your agent or skill supports it (e.g., via default parameters or a more advanced setup). - -```python -agent = OpenAIAgent( - dev_name="GreetingBot", - system_query="You are a polite bot. If a user asks for a greeting, use your GreeterSkill.", - skills=GreeterSkill, # Pass the GreeterSkill CLASS - # The agent will instantiate GreeterSkill. - # If the skill had required __init__ args not provided by the agent automatically, - # this direct class passing might be insufficient without further agent logic - # or by passing a pre-configured instance (see SkillLibrary example). - # For simple skills like GreeterSkill with defaults or optional args, this works well. - model_name="gpt-4o" -) -``` -In this setup, when the `GreetingBot` agent decides to use the `GreeterSkill`, it will instantiate it. If the `GreeterSkill` were to be instantiated by the agent with a specific `greeting_message`, the agent's design would need to support passing such parameters during skill instantiation. - -#### 2. Using a `SkillLibrary` Instance with an Agent - -Define the SkillLibrary and any skills it will manage in its collection: -```python -class MovementSkillsLibrary(SkillLibrary): - """A specialized skill library containing movement and navigation related skills.""" - - def __init__(self, robot=None): - super().__init__() - self._robot = robot - - def initialize_skills(self, robot=None): - """Initialize all movement skills with the robot instance.""" - if robot: - self._robot = robot - - if not self._robot: - raise ValueError("Robot instance is required to initialize skills") - - # Initialize with all movement-related skills - self.add(Navigate(robot=self._robot)) - self.add(NavigateToGoal(robot=self._robot)) - self.add(FollowHuman(robot=self._robot)) - self.add(NavigateToObject(robot=self._robot)) - self.add(GetPose(robot=self._robot)) # Position tracking skill -``` - -Note the addision of initialized skills added to this collection above. - -Proceed to use this skill library in an Agent: - -Finally, in your main application code: -```python -# 1. Create an instance of your custom skill library, configured with the robot -my_movement_skills = MovementSkillsLibrary(robot=robot_instance) - -# 2. Pass this library INSTANCE to the agent -performing_agent = OpenAIAgent( - dev_name="ShowBot", - system_query="You are a show robot. Use your skills as directed.", - skills=my_movement_skills, # Pass the configured SkillLibrary INSTANCE - model_name="gpt-4o" -) -``` - -### Unitree Test Files -- **`tests/run_go2_ros.py`**: Tests `UnitreeROSControl(ROSControl)` initialization in `UnitreeGo2(Robot)` via direct function calls `robot.move()` and `robot.webrtc_req()` -- **`tests/simple_agent_test.py`**: Tests a simple zero-shot class `OpenAIAgent` example -- **`tests/unitree/test_webrtc_queue.py`**: Tests `ROSCommandQueue` via a 20 back-to-back WebRTC requests to the robot -- **`tests/test_planning_agent_web_interface.py`**: Tests a simple two-stage `PlanningAgent` chained to an `ExecutionAgent` with backend FastAPI interface. -- **`tests/test_unitree_agent_queries_fastapi.py`**: Tests a zero-shot `ExecutionAgent` with backend FastAPI interface. - -## Documentation - -For detailed documentation, please visit our [documentation site](#) (Coming Soon). - -## Contributing +For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. -## License - -This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. - ## Acknowledgments Huge thanks to! @@ -492,11 +177,11 @@ Huge thanks to! - @legion1581 for his work on the [Unitree Go2 WebRTC Connect](https://github.com/legion1581/go2_webrtc_connect) from which we've pulled the ```Go2WebRTCConnection``` class and other types for seamless WebRTC-only integration with DimOS. - @tfoldi for the webrtc_req integration via Unitree Go2 ROS2 SDK, which allows for seamless usage of Unitree WebRTC control primitives with DimOS. +# License + +DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. + ## Contact - GitHub Issues: For bug reports and feature requests - Email: [build@dimensionalOS.com](mailto:build@dimensionalOS.com) - -## Known Issues -- Agent() failure to execute Nav2 action primitives (move, reverse, spinLeft, spinRight) is almost always due to the internal ROS2 collision avoidance, which will sometimes incorrectly display obstacles or be overly sensitive. Look for ```[behavior_server]: Collision Ahead - Exiting DriveOnHeading``` in the ROS logs. Reccomend restarting ROS2 or moving robot from objects to resolve. -- ```docker-compose up --build``` does not fully initialize the ROS2 environment due to ```std::bad_alloc``` errors. This will occur during continuous docker development if the ```docker-compose down``` is not run consistently before rebuilding and/or you are on a machine with less RAM, as ROS is very memory intensive. Reccomend running to clear your docker cache/images/containers with ```docker system prune``` and rebuild. diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md new file mode 100644 index 0000000000..0a3e2ceaf5 --- /dev/null +++ b/docs/concepts/blueprints.md @@ -0,0 +1,319 @@ +# Blueprints + +Blueprints (`ModuleBlueprint`) are instructions for how to initialize a `Module`. + +You don't typically want to run a single module, so multiple blueprints are handled together in `ModuleBlueprintSet`. + +You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: + +```python +blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') +``` + +But the same thing can be acomplished more succinctly as: + +```python +connection = ConnectionModule.blueprint +``` + +Now you can create the blueprint with: + +```python +blueprint = connection('arg1', 'arg2', kwarg='value') +``` + +## Linking blueprints + +You can link multiple blueprints together with `autoconnect`: + +```python +blueprint = autoconnect( + module1(), + module2(), + module3(), +) +``` + +`blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: + +```python +expanded_blueprint = autoconnect( + blueprint, + module4(), + module5(), +) +``` + +Blueprints are frozen data classes, and `autoconnect()` always constructs an expanded blueprint so you never have to worry about changes in one affecting the other. + +### Duplicate module handling + +If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: + +```python +blueprint = autoconnect( + module_a(arg1=1), + module_b(), + module_a(arg1=2), # This one is used, the first is discarded +) +``` + +This is so you can "inherit" from one blueprint but override something you need to change. + +## How transports are linked + +Imagine you have this code: + +```python +class ModuleA(Module): + image: Out[Image] + start_explore: Out[Bool] + +class ModuleB(Module): + image: In[Image] + begin_explore: In[Bool] + +module_a = partial(create_module_blueprint, ModuleA) +module_b = partial(create_module_blueprint, ModuleB) + +autoconnect(module_a(), module_b()) +``` + +Connections are linked based on `(property_name, object_type)`. In this case `('image', Image)` will be connected between the two modules, but `begin_explore` will not be linked to `start_explore`. + +## Topic names + +By default, the name of the property is used to generate the topic name. So for `image`, the topic will be `/image`. + +The property name is used only if it's unique. If two modules have the same property name with different types, then both get a random topic such as `/SGVsbG8sIFdvcmxkI`. + +If you don't like the name you can always override it like in the next section. + +## Which transport is used? + +By default `LCMTransport` is used if the object supports `lcm_encode`. If it doesn't `pLCMTransport` is used (meaning "pickled LCM"). + +You can override transports with the `transports` method. It returns a new blueprint in which the override is set. + +```python +blueprint = autoconnect(...) +expanded_blueprint = autoconnect(blueprint, ...) +blueprint = blueprint.transports({ + ("image", Image): pSHMTransport( + "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE + ), + ("start_explore", Bool): pLCMTransport(), +}) +``` + +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. + +## Remapping connections + +Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: + +```python +class ConnectionModule(Module): + color_image: Out[Image] # Outputs on 'color_image' + +class ProcessingModule(Module): + rgb_image: In[Image] # Expects input on 'rgb_image' + +# Without remapping, these wouldn't connect automatically +# With remapping, color_image is renamed to rgb_image +blueprint = ( + autoconnect( + ConnectionModule.blueprint(), + ProcessingModule.blueprint(), + ) + .remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), + ]) +) +``` + +After remapping: +- The `color_image` output from `ConnectionModule` is treated as `rgb_image` +- It automatically connects to any module with an `rgb_image` input of type `Image` +- The topic name becomes `/rgb_image` instead of `/color_image` + +If you want to override the topic, you still have to do it manually: + +```python +blueprint +.remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), +]) +.transports({ + ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), +}) +``` + +## Overriding global configuration. + +Each module can optionally take a `global_config` option in `__init__`. E.g.: + +```python +class ModuleA(Module): + + def __init__(self, global_config: GlobalConfig | None = None): + ... +``` + +The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: + +```python +blueprint = blueprint.global_config(n_dask_workers=8) +``` + +## Calling the methods of other modules + +Imagine you have this code: + +```python +class ModuleA(Module): + + @rpc + def get_time(self) -> str: + ... + +class ModuleB(Module): + def request_the_time(self) -> None: + ... +``` + +And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. + +To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. + +```python +class ModuleB(Module): + rpc_calls: list[str] = [ + "ModuleA.get_time", + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("ModuleA.get_time") + print(get_time_rpc()) +``` + +You can also request multiple methods at a time: + +```python +method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +``` + +## Alternative RPC calls + +There is an alternative way of receiving RPC methods. It is useful when you want to perform an action at the time you receive the RPC methods. + +You can use it by defining a method like `set__`: + +```python +class ModuleB(Module): + @rpc # Note that it has to be an rpc method. + def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: + self._get_time = rpc_call + self._get_time.set_rpc(self.rpc) + + def request_the_time(self) -> None: + print(self._get_time()) +``` + +Note that `RpcCall.rpc` does not serialize, so you have to set it to the one from the module with `rpc_call.set_rpc(self.rpc)` + +## Calling an interface + +In the previous examples, you can only call methods in a module called `ModuleA`. But what if you want to deploy an alternative module in your blueprint? + +You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. + +```python +class TimeInterface(ABC): + @abstractmethod + def get_time(self): ... + +class ProperTime(TimeInterface): + def get_time(self): + return "13:00" + +class BadTime(TimeInterface): + def get_time(self): + return "01:00 PM" + + +class ModuleB(Module): + rpc_calls: list[str] = [ + "TimeInterface.get_time", # TimeInterface instead of ProperTime or BadTime + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("TimeInterface.get_time") + print(get_time_rpc()) +``` + +The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: + +```python +blueprint = autoconnect( + ProperTime.blueprint(), + # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time + ModuleB.blueprint(), +) +``` + +If both are deployed, the blueprint will throw an error because it's ambiguous. + +## Defining skills + +Skills have to be registered with `AgentSpec.register_skills(self)`. + +```python +class SomeSkill(Module): + + @skill + def some_skill(self) -> None: + ... + + @rpc + def set_AgentSpec_register_skills(self, register_skills: RpcCall) -> None: + register_skills.set_rpc(self.rpc) + register_skills(RPCClient(self, self.__class__)) + + # The agent is just interested in the `@skill` methods, so you'll need this if your class + # has things that cannot be pickled. + def __getstate__(self): + pass + def __setstate__(self, _state): + pass +``` + +Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: + +```python +class SomeSkill(SkillModule): + + @skill + def some_skill(self) -> None: + ... +``` + +## Building + +All you have to do to build a blueprint is call: + +```python +module_coordinator = blueprint.build(global_config=config) +``` + +This returns a `ModuleCoordinator` instance that manages all deployed modules. + +### Running and shutting down + +You can block the thread until it exits with: + +```python +module_coordinator.loop() +``` + +This will wait for Ctrl+C and then automatically stop all modules and clean up resources. diff --git a/docs/development/README.md b/docs/development/README.md new file mode 100644 index 0000000000..b2dbd89382 --- /dev/null +++ b/docs/development/README.md @@ -0,0 +1,282 @@ +# Development Environment Guide + +1. How to setup your system (pick one: system install, nix flake + direnv, pure nix flake) +2. How to hack on DimOS +3. How to make a PR + +# 1. Setup + +All the tools below are optional and for your convenience. If you can get the code running on temple OS with a package manager you wrote yourself, all the power to you. + +--- + +## Setup Option A: System Install + +### Why pick this setup? (pros/cons/when-to-use) + +* Downside: not reliable, mutates your global system, causing (and receiving) side effects +* Upside: Often good for a quick hack or exploring +* Upside: Sometimes easier for CUDA/GPU acceleration +* Use when: you understand system package management (arch linux user) or you don't care about making changes to your system + +### How to setup DimOS + +```bash +# System dependencies + +# On Ubuntu 22.04 or 24.04 +if [ "$OSTYPE" = "linux-gnu" ]; then + sudo apt-get update + sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev pre-commit +# On macOS (12.6 or newer) +elif [ "$(uname)" = "Darwin" ]; then + # install homebrew + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + # install dependencies + brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python pre-commit +fi + +# install uv for python +curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + + +# create & activate a virtualenv (needed for dimos) +uv venv && . .venv/bin/activate + +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' + +# setup pre-commit +pre-commit install + +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features: + +- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector +- **contact_graspnet_pytorch**: Required for robotic grasp prediction + +You can install them with: + +```bash +uv add git+https://github.com/openai/CLIP.git +uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git +uv add git+https://github.com/facebookresearch/detectron2.git +``` + + + + +## Setup Option B: Nix Flake + direnv + +### Why pick this setup? (pros/cons/when-to-use) + +* Upside: Faster and more reliable than Dev Containers (no emulation) +* Upside: Nearly as isolated as Docker, but has full hardware access (CUDA, Webcam, networking) +* Downside: Not hard, but you need to install/understand [direnv](https://direnv.net/) (which you probably should do anyway) +* Downside: Nix is not user-friendly (IDE integration is not as good as Dev Containers) +* Use when: you need reliability and don't mind a one-time startup delay + +### Quickstart + +Install and activate [direnv](https://direnv.net/). + +```sh +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix .envrc +cp .envrc.nix .envrc +# this is going to take a while +direnv allow +direnv reload +direnv status + +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +## Setup Option C: Nix Flake - Isolated/Reliable + +### Why pick this setup? (pros/cons/when-to-use) + +* Use when: you need absolute reliability (use this if you want it to work first try) and don't mind a startup delay +* Upside: Doesn't need direnv, and has most of the other benefits of Nix +* Downside: Have to manually enter the environment (like `./venv/bin/activate` but slower) +* Upside: If you're you're using a basic shell, you'll get a nicely customized shell +* Downside: If you have hyper-customized your shell (fish, riced zsh, etc), you'll have to deal with someone else's preferences +* Downside: Your vibe coding agent will basically be unable to run tests for you (they don't understand how to enter the environment) + +### Quickstart + +```sh +# Install Nix +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +# make sure flakes are enabled +mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" + +# this allows getting large files on-demand +export GIT_LFS_SKIP_SMUDGE=1 +git clone -b main git@github.com:dimensionalOS/dimos.git +cd dimos + +# activate the nix development shell +nix develop '.#isolated' +``` + +Once inside the shell, run: + +```sh +# create virtualenv (needed for dimos) +uv venv && . .venv/bin/activate +# install dimos's python package with everything enabled +uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' +# test the install (takes about 3 minutes) +uv run pytest dimos +``` + +
+
+ +# 2. How to Hack on DimOS + +## Debugging + +Enable maximum logging: by adding `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1` as a prefix to the command. For example: + +```bash +DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos run unitree-go2 +``` + +This will save the rerun data to `rerun.json` in the current directory. + +## Where is `` located? (Architecture) + + +* If you want to add a `dimos run ` command see [dimos_run.md](/dimos/robot/cli/README.md) +* If you want to add a camera driver see [depth_camera_integration.md](/docs/depth_camera_integration.md) + +* For edits to manipulation see [manipulation.md](/dimos/hardware/manipulators/README.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) +* `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. +* `dimos/robot/`: Robot-specific modules live here. +* `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. +* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. +* `dimos/dashboard/`: Contains code related to visualization. +* `dimos/protocol/`: Defines low level stuff for communication between modules. +* See `dimos/` for the remainder + +## Testing + +We use both pytest and manual testing. + +```sh +pytest # run all tests at or below the current directory +``` + +### Testing Cheatsheet + +| Action | Command | +| --------------------------- | ---------------------------- | +| Run tests in current path | `pytest` | +| Filter tests by name | `pytest -k ""` | +| Enable stdout in tests | `pytest -s` | +| Run tagged tests | `pytest -m ` | +| Run all pre-commit hooks | `pre-commit run --all-files` | + +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) + +You can enable a tag by selecting -m - these are configured in `./pyproject.toml` + + +# 3. How to Make a PR +- Open the PR against the `dev` branch (not `main`) +- **No matter what, provide a one or few-lines that, when run, let a reviewer run the main path of the code you modified** (assuming you changed functional python code) +- If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. + +- Less changed files = better +- If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. +- We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). +- Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. +- If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). +- So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. +- If you're a new hire at DimOS: + - Smaller PR's are better. + - Only open a PR when you're okay with us spending AI tokens reviewing it (don't just open a trash PR and then fix it, wait till the code is mostly done) + - If there are 3 highly-intertwined bugs, make 3 PRs, not 1 PR. Yes it is more dev work, but review time is the bottleneck (not dev time). One line PR's are the easiest thing to review. + - When the AI (currently Greptile) comments on the code, respond. Sometimes Greptile is dumb as rocks but, as a reviewer, it's nice to see a finished conversation. + - Did we mention smaller PR's are better? From 26c69c2caa43c2aa2155b3e6d36e572c57553ebd Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 07:15:03 -0600 Subject: [PATCH 041/136] rice the readme --- README.md | 56 +++++++++++++++++++++++++++---------------------------- 1 file changed, 27 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 4dbfc8cac3..20ee5fd4db 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,36 @@ - - + banner_bordered_trimmed +

The Open-Source Framework for Robotic Intelligence

+ +
+ +[![Discord](https://img.shields.io/discord/1341146487186391173?style=flat-square&logo=discord&logoColor=white&label=Discord&color=5865F2)](https://discord.gg/8m6HMArf) +[![Stars](https://img.shields.io/github/stars/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/stargazers) +[![Forks](https://img.shields.io/github/forks/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/fork) +[![Contributors](https://img.shields.io/github/contributors/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/graphs/contributors) -# The Dimensional Framework -*The universal framework for AI-native generalist robotics* +

+ Key Features • + How To Use • + Contributing • + Credits • + License +

-## What is Dimensional? + -#### Warning: This is a pre-release version +# What is Dimensional? +> \[!NOTE] +> +> **Active Beta: Expect Breaking Changes** -Dimensional is an open-source framework for adding customized general intelligence to robots. DimOS allows AI agents to call tools/functions (skills), read sensor/state data directly, and use them to produce robust emergent behavior. DimOS is both specification based (use any programming language) and a python-first library that works well with (and without) [ROS](https://www.ros.org/). The python library comes with a rich set of integrations; spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. +DimOS is both specification based (any programming language) and a python-first library for controlling robots. Works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. + +The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. # How do I get started? -## Installation +### Installation #### Details / Requirements @@ -163,13 +166,13 @@ There are several tools: - [Skills](/docs/concepts/blueprints.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools -# Contributing / Building From Source +## Contributing / Building From Source For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. -## Acknowledgments +# Acknowledgments Huge thanks to! - The Roboverse Community and their unitree-specific help. Check out their [Discord](https://discord.gg/HEXNMCNhEh). @@ -180,8 +183,3 @@ Huge thanks to! # License DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. - -## Contact - -- GitHub Issues: For bug reports and feature requests -- Email: [build@dimensionalOS.com](mailto:build@dimensionalOS.com) From ec5180f0f73b3ecbebe09b76971aa12f87f7aad5 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 07:18:46 -0600 Subject: [PATCH 042/136] grammar --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 20ee5fd4db..8bda3d75c1 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ > > **Active Beta: Expect Breaking Changes** -DimOS is both specification based (any programming language) and a python-first library for controlling robots. Works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. +DimOS is both a specification based (any programming language) framework and a python-first library for controlling robots. DimOS works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From c737ff2e308c43856c716c54168ebb37b0d91470 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 07:26:31 -0600 Subject: [PATCH 043/136] formatting --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8bda3d75c1..5d8b25020b 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,19 @@
- banner_bordered_trimmed + banner_bordered_trimmed

The Open-Source Framework for Robotic Intelligence

+
[![Discord](https://img.shields.io/discord/1341146487186391173?style=flat-square&logo=discord&logoColor=white&label=Discord&color=5865F2)](https://discord.gg/8m6HMArf) [![Stars](https://img.shields.io/github/stars/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/stargazers) [![Forks](https://img.shields.io/github/forks/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/fork) [![Contributors](https://img.shields.io/github/contributors/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/graphs/contributors) +
+![Nix](https://img.shields.io/badge/Nix-flakes-5277C3?style=flat-square&logo=NixOS&logoColor=white) +![NixOS](https://img.shields.io/badge/NixOS-supported-5277C3?style=flat-square&logo=NixOS&logoColor=white) +![CUDA](https://img.shields.io/badge/CUDA-12.x-76B900?style=flat-square&logo=nvidia&logoColor=white) +[![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square&logo=docker&logoColor=white)](https://www.docker.com/)

Key Features • @@ -19,11 +25,12 @@

-# What is Dimensional? > \[!NOTE] > > **Active Beta: Expect Breaking Changes** +# What is Dimensional? + DimOS is both a specification based (any programming language) framework and a python-first library for controlling robots. DimOS works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From e0fe4a7a4d7d656d4caac2df0115e92b21882eeb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 05:39:36 -0800 Subject: [PATCH 044/136] fix examples --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 5d8b25020b..eedd182b3b 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@
-[![Discord](https://img.shields.io/discord/1341146487186391173?style=flat-square&logo=discord&logoColor=white&label=Discord&color=5865F2)](https://discord.gg/8m6HMArf) +[![Discord](https://img.shields.io/discord/1341146487186391173?style=flat-square&logo=discord&logoColor=white&label=Discord&color=5865F2)](https://discord.gg/8m6HMArf) [![Stars](https://img.shields.io/github/stars/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/stargazers) [![Forks](https://img.shields.io/github/forks/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/fork) [![Contributors](https://img.shields.io/github/contributors/dimensionalOS/dimos?style=flat-square)](https://github.com/dimensionalOS/dimos/graphs/contributors) @@ -31,7 +31,7 @@ # What is Dimensional? -DimOS is both a specification based (any programming language) framework and a python-first library for controlling robots. DimOS works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. +DimOS is both a specification based (any programming language) framework and a python-first library for controlling robots. DimOS works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. @@ -102,7 +102,7 @@ Simple camera activation (save this as a python file and run it): ```py from dimos.core.blueprints import autoconnect -from dimos.hardware.camera.module import CameraModule +from dimos.hardware.sensors.camera.module import CameraModule if __name__ == "__main__": autoconnect( @@ -115,9 +115,9 @@ Write your own custom module: ```py from dimos.core.blueprints import autoconnect -from dimos.core import In, Module, pSHMTransport +from dimos.core import In, Out, Module from dimos.core.core import rpc -from dimos.hardware.camera.module import CameraModule +from dimos.hardware.sensors.camera.module import CameraModule from dimos.msgs.sensor_msgs import Image from reactivex.disposable import Disposable From f8681b2fdcc7f4ed8e98d26774d49970c9c04252 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 05:54:54 -0800 Subject: [PATCH 045/136] change links to reduce change count --- README.md | 6 +- docs/concepts/blueprints.md | 319 ------------------------------------ 2 files changed, 3 insertions(+), 322 deletions(-) delete mode 100644 docs/concepts/blueprints.md diff --git a/README.md b/README.md index eedd182b3b..d0c6097741 100644 --- a/README.md +++ b/README.md @@ -168,9 +168,9 @@ if __name__ == "__main__": There are several tools: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. -- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other -- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) -- [Skills](/docs/concepts/blueprints.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other +- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools ## Contributing / Building From Source diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md deleted file mode 100644 index 0a3e2ceaf5..0000000000 --- a/docs/concepts/blueprints.md +++ /dev/null @@ -1,319 +0,0 @@ -# Blueprints - -Blueprints (`ModuleBlueprint`) are instructions for how to initialize a `Module`. - -You don't typically want to run a single module, so multiple blueprints are handled together in `ModuleBlueprintSet`. - -You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: - -```python -blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') -``` - -But the same thing can be acomplished more succinctly as: - -```python -connection = ConnectionModule.blueprint -``` - -Now you can create the blueprint with: - -```python -blueprint = connection('arg1', 'arg2', kwarg='value') -``` - -## Linking blueprints - -You can link multiple blueprints together with `autoconnect`: - -```python -blueprint = autoconnect( - module1(), - module2(), - module3(), -) -``` - -`blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: - -```python -expanded_blueprint = autoconnect( - blueprint, - module4(), - module5(), -) -``` - -Blueprints are frozen data classes, and `autoconnect()` always constructs an expanded blueprint so you never have to worry about changes in one affecting the other. - -### Duplicate module handling - -If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: - -```python -blueprint = autoconnect( - module_a(arg1=1), - module_b(), - module_a(arg1=2), # This one is used, the first is discarded -) -``` - -This is so you can "inherit" from one blueprint but override something you need to change. - -## How transports are linked - -Imagine you have this code: - -```python -class ModuleA(Module): - image: Out[Image] - start_explore: Out[Bool] - -class ModuleB(Module): - image: In[Image] - begin_explore: In[Bool] - -module_a = partial(create_module_blueprint, ModuleA) -module_b = partial(create_module_blueprint, ModuleB) - -autoconnect(module_a(), module_b()) -``` - -Connections are linked based on `(property_name, object_type)`. In this case `('image', Image)` will be connected between the two modules, but `begin_explore` will not be linked to `start_explore`. - -## Topic names - -By default, the name of the property is used to generate the topic name. So for `image`, the topic will be `/image`. - -The property name is used only if it's unique. If two modules have the same property name with different types, then both get a random topic such as `/SGVsbG8sIFdvcmxkI`. - -If you don't like the name you can always override it like in the next section. - -## Which transport is used? - -By default `LCMTransport` is used if the object supports `lcm_encode`. If it doesn't `pLCMTransport` is used (meaning "pickled LCM"). - -You can override transports with the `transports` method. It returns a new blueprint in which the override is set. - -```python -blueprint = autoconnect(...) -expanded_blueprint = autoconnect(blueprint, ...) -blueprint = blueprint.transports({ - ("image", Image): pSHMTransport( - "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE - ), - ("start_explore", Bool): pLCMTransport(), -}) -``` - -Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. - -## Remapping connections - -Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: - -```python -class ConnectionModule(Module): - color_image: Out[Image] # Outputs on 'color_image' - -class ProcessingModule(Module): - rgb_image: In[Image] # Expects input on 'rgb_image' - -# Without remapping, these wouldn't connect automatically -# With remapping, color_image is renamed to rgb_image -blueprint = ( - autoconnect( - ConnectionModule.blueprint(), - ProcessingModule.blueprint(), - ) - .remappings([ - (ConnectionModule, 'color_image', 'rgb_image'), - ]) -) -``` - -After remapping: -- The `color_image` output from `ConnectionModule` is treated as `rgb_image` -- It automatically connects to any module with an `rgb_image` input of type `Image` -- The topic name becomes `/rgb_image` instead of `/color_image` - -If you want to override the topic, you still have to do it manually: - -```python -blueprint -.remappings([ - (ConnectionModule, 'color_image', 'rgb_image'), -]) -.transports({ - ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), -}) -``` - -## Overriding global configuration. - -Each module can optionally take a `global_config` option in `__init__`. E.g.: - -```python -class ModuleA(Module): - - def __init__(self, global_config: GlobalConfig | None = None): - ... -``` - -The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: - -```python -blueprint = blueprint.global_config(n_dask_workers=8) -``` - -## Calling the methods of other modules - -Imagine you have this code: - -```python -class ModuleA(Module): - - @rpc - def get_time(self) -> str: - ... - -class ModuleB(Module): - def request_the_time(self) -> None: - ... -``` - -And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. - -To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. - -```python -class ModuleB(Module): - rpc_calls: list[str] = [ - "ModuleA.get_time", - ] - - def request_the_time(self) -> None: - get_time_rpc = self.get_rpc_calls("ModuleA.get_time") - print(get_time_rpc()) -``` - -You can also request multiple methods at a time: - -```python -method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") -``` - -## Alternative RPC calls - -There is an alternative way of receiving RPC methods. It is useful when you want to perform an action at the time you receive the RPC methods. - -You can use it by defining a method like `set__`: - -```python -class ModuleB(Module): - @rpc # Note that it has to be an rpc method. - def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: - self._get_time = rpc_call - self._get_time.set_rpc(self.rpc) - - def request_the_time(self) -> None: - print(self._get_time()) -``` - -Note that `RpcCall.rpc` does not serialize, so you have to set it to the one from the module with `rpc_call.set_rpc(self.rpc)` - -## Calling an interface - -In the previous examples, you can only call methods in a module called `ModuleA`. But what if you want to deploy an alternative module in your blueprint? - -You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. - -```python -class TimeInterface(ABC): - @abstractmethod - def get_time(self): ... - -class ProperTime(TimeInterface): - def get_time(self): - return "13:00" - -class BadTime(TimeInterface): - def get_time(self): - return "01:00 PM" - - -class ModuleB(Module): - rpc_calls: list[str] = [ - "TimeInterface.get_time", # TimeInterface instead of ProperTime or BadTime - ] - - def request_the_time(self) -> None: - get_time_rpc = self.get_rpc_calls("TimeInterface.get_time") - print(get_time_rpc()) -``` - -The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: - -```python -blueprint = autoconnect( - ProperTime.blueprint(), - # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time - ModuleB.blueprint(), -) -``` - -If both are deployed, the blueprint will throw an error because it's ambiguous. - -## Defining skills - -Skills have to be registered with `AgentSpec.register_skills(self)`. - -```python -class SomeSkill(Module): - - @skill - def some_skill(self) -> None: - ... - - @rpc - def set_AgentSpec_register_skills(self, register_skills: RpcCall) -> None: - register_skills.set_rpc(self.rpc) - register_skills(RPCClient(self, self.__class__)) - - # The agent is just interested in the `@skill` methods, so you'll need this if your class - # has things that cannot be pickled. - def __getstate__(self): - pass - def __setstate__(self, _state): - pass -``` - -Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: - -```python -class SomeSkill(SkillModule): - - @skill - def some_skill(self) -> None: - ... -``` - -## Building - -All you have to do to build a blueprint is call: - -```python -module_coordinator = blueprint.build(global_config=config) -``` - -This returns a `ModuleCoordinator` instance that manages all deployed modules. - -### Running and shutting down - -You can block the thread until it exits with: - -```python -module_coordinator.loop() -``` - -This will wait for Ctrl+C and then automatically stop all modules and clean up resources. From 8c4242abb783c4879dd45559835edb0488eae808 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:27:02 -0800 Subject: [PATCH 046/136] improve wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d0c6097741..811559595d 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ # What is Dimensional? -DimOS is both a specification based (any programming language) framework and a python-first library for controlling robots. DimOS works with (and without) [ROS](https://www.ros.org/), with design that enables AI agents to call tools/functions (skills), read sensor/state data directly, and generate complex emergent behaviors. +DimOS is both a specification-based, language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From ed50ba78eff022d492beaa99c848479ad575732f Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:28:06 -0800 Subject: [PATCH 047/136] wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 811559595d..c9774e9565 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ DimOS is both a specification-based, language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. -The python library comes with a rich set of integrations; visualization, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. +The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. # How do I get started? From abe476c3177bc340ea545de3654f3e71c8224dfa Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:31:42 -0800 Subject: [PATCH 048/136] remove acknowledgements --- README.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/README.md b/README.md index c9774e9565..bd9d442df2 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Key Features • How To Use • Contributing • - Credits • + License

@@ -179,14 +179,6 @@ For development, we optimize for flexibility—whether you love Docker, Nix, or We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. -# Acknowledgments - -Huge thanks to! -- The Roboverse Community and their unitree-specific help. Check out their [Discord](https://discord.gg/HEXNMCNhEh). -- @abizovnuralem for his work on the [Unitree Go2 ROS2 SDK](https://github.com/abizovnuralem/go2_ros2_sdk) we integrate with for DimOS. -- @legion1581 for his work on the [Unitree Go2 WebRTC Connect](https://github.com/legion1581/go2_webrtc_connect) from which we've pulled the ```Go2WebRTCConnection``` class and other types for seamless WebRTC-only integration with DimOS. -- @tfoldi for the webrtc_req integration via Unitree Go2 ROS2 SDK, which allows for seamless usage of Unitree WebRTC control primitives with DimOS. - # License DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. From 94178d24b8881d93cc9c076e840899bec20575c4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:41:31 -0800 Subject: [PATCH 049/136] improve the humancli example --- README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index bd9d442df2..f9f7541b9e 100644 --- a/README.md +++ b/README.md @@ -85,13 +85,17 @@ dimos --simulation run unitree-go2 #### Have it controlled by AI! -```bash +WARNING: This is a demo showing the **connection** between AI and robotic control -- not a demo of a super-intelligent AI. Be ready to physically prevent your robot from taking dumb physical actions. + +```sh export OPENAI_API_KEY= dimos run unitree-go2-agentic -# open the following in a different terminal tab to tell it where to go -# Warning!: make sure to watch the bot, this is a pre-release it will run into stuff -# and get tangled/trapped -# ex: tell it to explore the room, then tell it to go to where it can see a door +``` + +Then open this chat in a separate terminal (don't forget to activate the venv) +```sh +# ex: tell it to explore the room +# then tell it to go to something such as "go to the door" humancli ``` From 65609829979b88e08f62cd28976892de49167840 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:42:21 -0800 Subject: [PATCH 050/136] formatting --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index f9f7541b9e..a2b067a417 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,7 @@

Key Features • How To Use • - Contributing • - - License + Contributing • License

From d5240122d9c8b481b0cd585de11301684fbd303a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:43:28 -0800 Subject: [PATCH 051/136] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a2b067a417..ab49bc82ad 100644 --- a/README.md +++ b/README.md @@ -37,8 +37,6 @@ The python library comes with a rich set of integrations; visualizers, spatial r ### Installation -#### Details / Requirements - - Linux, tested on Ubuntu 22.04, 24.04 - MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` @@ -65,7 +63,9 @@ dimos --replay run unitree-go2 -#### Get it working on a real physical robot! +### Usage + +#### Get it working on a physical robot! ```sh export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE From d57399264b3a7dee06015a3f4cd8eef2d2c82704 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:45:28 -0800 Subject: [PATCH 052/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ab49bc82ad..f425e7755a 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ The python library comes with a rich set of integrations; visualizers, spatial r ### Installation -- Linux, tested on Ubuntu 22.04, 24.04 +- Linux is supported, with tests being performed on Ubuntu 22.04 and 24.04 - MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` From 4c47dcbbbf330e2d1bff6585178d99cf6be7ee54 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:49:30 -0800 Subject: [PATCH 053/136] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f425e7755a..6b174998d9 100644 --- a/README.md +++ b/README.md @@ -90,10 +90,10 @@ export OPENAI_API_KEY= dimos run unitree-go2-agentic ``` -Then open this chat in a separate terminal (don't forget to activate the venv) +After running that, open a new terminal (activate the venv again `source .venv/bin/activate`) and run the following command to start controlling the agent. ```sh # ex: tell it to explore the room -# then tell it to go to something such as "go to the door" +# then tell it to go to somethingn it saw such as "go to the door" humancli ``` From 0c4162adb68f2044dd2d58ed6ee73412c5ccc8bb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:52:47 -0800 Subject: [PATCH 054/136] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6b174998d9..a4e27fffd1 100644 --- a/README.md +++ b/README.md @@ -90,8 +90,10 @@ export OPENAI_API_KEY= dimos run unitree-go2-agentic ``` -After running that, open a new terminal (activate the venv again `source .venv/bin/activate`) and run the following command to start controlling the agent. +After running that, open a new terminal and run the following command to start a conversation with the agent that controls the robot. ```sh +# activate the venv in this new terminal +source .venv/bin/activate # ex: tell it to explore the room # then tell it to go to somethingn it saw such as "go to the door" humancli From f199869a47873438bf375d696a08997292385a2c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:54:44 -0800 Subject: [PATCH 055/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a4e27fffd1..7248ae08cb 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ export OPENAI_API_KEY= dimos run unitree-go2-agentic ``` -After running that, open a new terminal and run the following command to start a conversation with the agent that controls the robot. +After running that, open a new terminal and run the following to start giving instructions to the agent. ```sh # activate the venv in this new terminal source .venv/bin/activate From 1cca52ed65f6a3d632bdac298c8d4e052847532a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:56:46 -0800 Subject: [PATCH 056/136] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7248ae08cb..0029eb405f 100644 --- a/README.md +++ b/README.md @@ -94,8 +94,8 @@ After running that, open a new terminal and run the following to start giving in ```sh # activate the venv in this new terminal source .venv/bin/activate -# ex: tell it to explore the room -# then tell it to go to somethingn it saw such as "go to the door" +# after running the following, tell the agent "explore the room" +# then tell it to go to something it saw such as "go to the door" humancli ``` From a9198d072c21c679e12e282d9337bf94f72699a7 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 11:57:26 -0800 Subject: [PATCH 057/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0029eb405f..f320176f67 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ After running that, open a new terminal and run the following to start giving in ```sh # activate the venv in this new terminal source .venv/bin/activate -# after running the following, tell the agent "explore the room" +# after running the following, tell the agent "explore the room" # then tell it to go to something it saw such as "go to the door" humancli ``` From 4793eaac5d4ae9eb478bdba1eac0ff356ae3d2fc Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 12:02:50 -0800 Subject: [PATCH 058/136] Update README.md --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f320176f67..d1a037d80c 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,10 @@ After running that, open a new terminal and run the following to start giving in ```sh # activate the venv in this new terminal source .venv/bin/activate -# after running the following, tell the agent "explore the room" + +# Note: after running the next command, WAIT for the agent to connect before sending commands +# this will take a while the first time +# then tell the agent "explore the room" # then tell it to go to something it saw such as "go to the door" humancli ``` From 1c6dbed5087d970ffbef16f45659d526607befbb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 12:34:05 -0800 Subject: [PATCH 059/136] Update README.md --- README.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d1a037d80c..fe0b02e409 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,21 @@ if __name__ == "__main__": ).build().loop() ``` -### Note: Many More Examples in the [Examples Folder](./examples) +#### Note: Many More Examples in the [Examples Folder](./examples) + + + +### How does that example work? + +- Every module represents one process: all modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves** Don't mutate or share global vars inside a module. +- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects someone somewhere to give it a color image. The module is going to publish a grayscale image. +- The `autoconnect` ties everything together: + - The CameraModule has an output of `color_image` + - The Listener has an input of `color_image` + - Autoconnect puts them together, and checks that their types are compatible (both are of type `Image`) +- What about `@rpc`? + - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. + - The start/stop methods always need to be an rpc because they are called externally. # How does DimOS work conceptually? @@ -175,10 +189,10 @@ if __name__ == "__main__": There are several tools: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. -- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other -- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data) +- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. +- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). - [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). -- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. ## Contributing / Building From Source From cf7d06758bfc5249e5655140e73ef6f65b9687bd Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 12:38:19 -0800 Subject: [PATCH 060/136] remove print that never runs --- README.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index fe0b02e409..c4082db27c 100644 --- a/README.md +++ b/README.md @@ -52,13 +52,13 @@ curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin # # the command needs to download the replay file (2.4gb), which takes a bit -# OPTION 1: use without installing dimos -uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 - -# OPTION 2: install dimos in a virtualenv +# OPTION 1: install dimos in a virtualenv uv venv && . .venv/bin/activate uv pip install 'dimos[base,unitree]' dimos --replay run unitree-go2 + +# OPTION 2: if you want to test out dimos without installing run: +uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 ``` @@ -98,7 +98,7 @@ source .venv/bin/activate # Note: after running the next command, WAIT for the agent to connect before sending commands # this will take a while the first time # then tell the agent "explore the room" -# then tell it to go to something it saw such as "go to the door" +# then tell it to go to something it saw such as "go to the door" humancli ``` @@ -115,7 +115,6 @@ if __name__ == "__main__": autoconnect( CameraModule.blueprint() ).build().loop() - print("Webcam pipeline running. Press Ctrl+C to stop.") ``` Write your own custom module: @@ -173,7 +172,7 @@ if __name__ == "__main__": ### How does that example work? -- Every module represents one process: all modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves** Don't mutate or share global vars inside a module. +- Every module represents one process: all modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves** Don't mutate or share global vars inside a module. - At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects someone somewhere to give it a color image. The module is going to publish a grayscale image. - The `autoconnect` ties everything together: - The CameraModule has an output of `color_image` From 71aa141e9dbf3f63495f1bc57e23bd2280c2d6e1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 12:39:00 -0800 Subject: [PATCH 061/136] wording --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c4082db27c..b09869036f 100644 --- a/README.md +++ b/README.md @@ -95,10 +95,10 @@ After running that, open a new terminal and run the following to start giving in # activate the venv in this new terminal source .venv/bin/activate -# Note: after running the next command, WAIT for the agent to connect before sending commands -# this will take a while the first time +# Note: after running the next command, WAIT for the agent to connect +# (this will take a while the first time) # then tell the agent "explore the room" -# then tell it to go to something it saw such as "go to the door" +# then tell it to go to something, ex: "go to the door" humancli ``` From a593ed962d98e5ab52c829cb4e2566fd15e04429 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 12:45:44 -0800 Subject: [PATCH 062/136] example of getting module inputs/outputs --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index b09869036f..3bdbe8ea69 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,12 @@ if __name__ == "__main__": - The CameraModule has an output of `color_image` - The Listener has an input of `color_image` - Autoconnect puts them together, and checks that their types are compatible (both are of type `Image`) +- How can we see what In/Out streams are provided by a module? + - Open a python repl (e.g. `python`) + - Import the module, ex: `from dimos.hardware.sensors.camera.module import CameraModule` + - Print the module outputs: `print(CameraModule.module_info().outputs)` + - Print the module inputs: `print(CameraModule.module_info().inputs)` + - Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())` - What about `@rpc`? - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. - The start/stop methods always need to be an rpc because they are called externally. From 4c09ee4c742ccc669e1c5eb642451fc9633403ec Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:08:19 -0800 Subject: [PATCH 063/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b09869036f..413ada6727 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ # What is Dimensional? -DimOS is both a specification-based, language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. +DimOS is both a specification-based language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From a202a5dc4c485cd1b675c06a9d7052e8f78ee54d Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:08:39 -0800 Subject: [PATCH 064/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 413ada6727..eee4b629be 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ # What is Dimensional? -DimOS is both a specification-based language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. +DimOS is both a language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From 60e606028e6a3aa56bdb39c90b0f13f74cf28f68 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:09:18 -0800 Subject: [PATCH 065/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eee4b629be..1abbc1a931 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ # What is Dimensional? -DimOS is both a language-agnostic framework and a Python-first library for robot control. It works with or without ROS and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. +DimOS is both a language-agnostic framework and a Python-first library for robot control. It has optional ROS integration and is designed to let AI agents invoke tools (skills), directly access sensor and state data, and generate complex emergent behaviors. The python library comes with a rich set of integrations; visualizers, spatial reasoners, planners, simulators (mujoco, Isaac Sim, etc.), robot state/action primitives, and more. From 145a4dc8cdb20ce1e7b99252d56005cef150a061 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:10:52 -0800 Subject: [PATCH 066/136] Update README.md --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 1abbc1a931..91ffebe8bb 100644 --- a/README.md +++ b/README.md @@ -65,14 +65,7 @@ uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 ### Usage -#### Get it working on a physical robot! - -```sh -export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE -dimos run unitree-go2 -``` - -#### Get it working in an interactive simulation! +#### Control a robot in a simulation (no robot required) ```bash export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors @@ -81,6 +74,13 @@ dimos --simulation run unitree-go2 # open http://localhost:7779/command-center in your browser to control the robot movement ``` +#### Get it working on a physical robot! + +```sh +export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +dimos run unitree-go2 +``` + #### Have it controlled by AI! WARNING: This is a demo showing the **connection** between AI and robotic control -- not a demo of a super-intelligent AI. Be ready to physically prevent your robot from taking dumb physical actions. From 6b9f4b37da6dbf6ff233c4343d893d9f8507ad2a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:11:49 -0800 Subject: [PATCH 067/136] - --- README.md | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 3bdbe8ea69..0346c4b343 100644 --- a/README.md +++ b/README.md @@ -104,8 +104,9 @@ humancli # How do I use it as a library? +### Simple Camera Activation -Simple camera activation (save this as a python file and run it): +Assuming you have a webcam, save the following as a python file and run it: ```py from dimos.core.blueprints import autoconnect @@ -117,12 +118,13 @@ if __name__ == "__main__": ).build().loop() ``` -Write your own custom module: +### Write A Custom Module + +Lets convert the camera's image to grayscale. ```py from dimos.core.blueprints import autoconnect -from dimos.core import In, Out, Module -from dimos.core.core import rpc +from dimos.core import In, Out, Module, rpc from dimos.hardware.sensors.camera.module import CameraModule from dimos.msgs.sensor_msgs import Image @@ -130,9 +132,8 @@ from reactivex.disposable import Disposable class Listener(Module): # the CameraModule has an Out[Image] named "color_image" - # this module will only receive those messages if - # the names ("color_image") match, otherwise autoconnect - # will not be able to connect one module to another + # How do we know this? Just print(CameraModule.module_info().outputs) + # the name ("color_image") must match the CameraModule's output color_image: In[Image] = None grayscale_image: Out[Image] = None @@ -168,12 +169,11 @@ if __name__ == "__main__": #### Note: Many More Examples in the [Examples Folder](./examples) +### How do custom modules work? - -### How does that example work? - -- Every module represents one process: all modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves** Don't mutate or share global vars inside a module. -- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects someone somewhere to give it a color image. The module is going to publish a grayscale image. +- Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. +- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). + - Note: if you are a power user thinking "so streams must be statically declared?" the answer is no, there are ways to perform dynamic connections, but for type-checking and human sanity the creation of dynamic stream connections are under an advanced API and should be used as a last resort. - The `autoconnect` ties everything together: - The CameraModule has an output of `color_image` - The Listener has an input of `color_image` @@ -189,9 +189,11 @@ if __name__ == "__main__": - The start/stop methods always need to be an rpc because they are called externally. + + # How does DimOS work conceptually? -There are several tools: +There are several conceptual tools: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. - [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. @@ -199,6 +201,11 @@ There are several tools: - [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. +There are also many monitoring tools: +- Run `lcmspy` to see how fast messages are being published on streams +- Run `skillspy` to see how skills are being called, how long they are running, which are active, etc +- Run `agentspy` to see the agent's status over time + ## Contributing / Building From Source For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. From 9310f5d6fdfcc0c344fa24027333ebee941a49b5 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:36:12 -0800 Subject: [PATCH 068/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f4b2662edb..642f8b087a 100644 --- a/README.md +++ b/README.md @@ -169,7 +169,7 @@ if __name__ == "__main__": #### Note: Many More Examples in the [Examples Folder](./examples) -### How do custom modules work? +### How does that custom module work? - Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. - At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). From c438b3a886811f4f04edb797e7a06198d2f89243 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:37:30 -0800 Subject: [PATCH 069/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 642f8b087a..11396d4a77 100644 --- a/README.md +++ b/README.md @@ -169,7 +169,7 @@ if __name__ == "__main__": #### Note: Many More Examples in the [Examples Folder](./examples) -### How does that custom module work? +### How do custom modules work? (Example explanation) - Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. - At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). From d4ad93e3a9a87f008f336b1a8d248b30ed4d4df3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:38:07 -0800 Subject: [PATCH 070/136] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 11396d4a77..7208087873 100644 --- a/README.md +++ b/README.md @@ -169,7 +169,7 @@ if __name__ == "__main__": #### Note: Many More Examples in the [Examples Folder](./examples) -### How do custom modules work? (Example explanation) +### How do custom modules work? (Example breakdown) - Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. - At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). From bcfc5ede87bf8092cc05f5873e992b881da883f1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:38:58 -0800 Subject: [PATCH 071/136] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7208087873..5c4d4837e4 100644 --- a/README.md +++ b/README.md @@ -191,9 +191,9 @@ if __name__ == "__main__": -# How does DimOS work conceptually? +# How does DimOS work? -There are several conceptual tools: +Concepts: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. - [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. From b2682dced624edcb83e7a6c0023c2a948fb0ed47 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:41:11 -0800 Subject: [PATCH 072/136] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5c4d4837e4..b45451dc9b 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,7 @@ if __name__ == "__main__": - Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())` - What about `@rpc`? - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. + - Rpc methods get called using threads, meaning two rpc methods can be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. - The start/stop methods always need to be an rpc because they are called externally. @@ -194,7 +195,7 @@ if __name__ == "__main__": # How does DimOS work? Concepts: -- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are defined in python as classes. +- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. - [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. - [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). From 4b8ad01d45388f7bb8f9ac824f18b4710624d211 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 13:49:45 -0800 Subject: [PATCH 073/136] Update README.md --- docs/development/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index b2dbd89382..713755dc2f 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -12,9 +12,9 @@ All the tools below are optional and for your convenience. If you can get the co ## Setup Option A: System Install -### Why pick this setup? (pros/cons/when-to-use) +### Why pick this option? (pros/cons/when-to-use) -* Downside: not reliable, mutates your global system, causing (and receiving) side effects +* Downside: mutates your global system, causing (and receiving) side effects causes it to be unreliable * Upside: Often good for a quick hack or exploring * Upside: Sometimes easier for CUDA/GPU acceleration * Use when: you understand system package management (arch linux user) or you don't care about making changes to your system @@ -74,7 +74,7 @@ uv add git+https://github.com/facebookresearch/detectron2.git -- Less changed files = better -- If you know one of your code changes will "look weird" or be up for debate, open the github UI and add a graphical comment on that code. In that comment justify youraq choice and explaining downsides of alternatives. -- We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably have unit tests or good reason why not (ex: visualization stuff is hard to unit test so we don't). +- Open the PR against the `dev` branch (not `main`). +- **No matter what, provide a few-lines that, when run, let a reviewer test the main feature of the code you modified** (assuming you changed functional python code). +- Less changed files = better. +- If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. +- If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. +- We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. - If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - - Smaller PR's are better. - - Only open a PR when you're okay with us spending AI tokens reviewing it (don't just open a trash PR and then fix it, wait till the code is mostly done) + - Did we mention smaller PR's are better? Smaller PR's are better. + - Only open a PR when you're okay with us spending AI tokens reviewing it (don't open a half-done PR and then fix it, wait till the code is mostly done). - If there are 3 highly-intertwined bugs, make 3 PRs, not 1 PR. Yes it is more dev work, but review time is the bottleneck (not dev time). One line PR's are the easiest thing to review. - When the AI (currently Greptile) comments on the code, respond. Sometimes Greptile is dumb as rocks but, as a reviewer, it's nice to see a finished conversation. - - Did we mention smaller PR's are better? From 405f23a66a76230b91a0d97589daf42eaa73a0c4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:00:22 -0800 Subject: [PATCH 076/136] Update README.md --- docs/development/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index f1b8ce6271..466c259139 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -212,7 +212,6 @@ uv pip install -e '.[base,dev,manipulation,misc,unitree,drone]' uv run pytest dimos ``` -

# 2. How to Hack on DimOS From dbb26b12d82bf55c60904bfd2f803913826e427f Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:00:43 -0800 Subject: [PATCH 077/136] Update README.md --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index 466c259139..4ce253ac2e 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,4 +1,4 @@ -# Development Environment Guide +# Development Guide 1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) 2. [How to hack on DimOS](#2-how-to-hack-on-dimos) From 2e2ea1ed86268e8bb2c3b30aed2968a5c3267f28 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:01:45 -0800 Subject: [PATCH 078/136] Update README.md --- docs/development/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index 4ce253ac2e..7a6807442c 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,8 +1,8 @@ # Development Guide 1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) -2. [How to hack on DimOS](#2-how-to-hack-on-dimos) -4. [How to make a PR](#3-how-to-make-a-pr) +2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (find what you want to change) +4. [How to make a PR](#3-how-to-make-a-pr) (our expectations for a PR)
From b68eb9b74d9e3ece1dc62415cded71e01f1edf7a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:03:10 -0800 Subject: [PATCH 079/136] Update README.md --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index 7a6807442c..35af1005bd 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,7 +1,7 @@ # Development Guide 1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) -2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (find what you want to change) +2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (which files to edit, debugging help, etc) 4. [How to make a PR](#3-how-to-make-a-pr) (our expectations for a PR)
From 41a867ebc63f26a3e75ebbc8a86d3aa120c4704b Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:04:23 -0800 Subject: [PATCH 080/136] Update README.md --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index 35af1005bd..c93e9c863b 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -267,7 +267,7 @@ You can enable a tag by selecting -m - these are configured in `./pyp # 3. How to Make a PR - Open the PR against the `dev` branch (not `main`). -- **No matter what, provide a few-lines that, when run, let a reviewer test the main feature of the code you modified** (assuming you changed functional python code). +- **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. - If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. From 2575b617b89f2a1ca5765776d15102717afdeadd Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:06:12 -0800 Subject: [PATCH 081/136] Update README.md --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index c93e9c863b..832275ccbd 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -8,7 +8,7 @@ # 1. Setup -All the tools below are optional and for your convenience. If you can get the code running on temple OS with a package manager you wrote yourself, all the power to you. +All the setup options are for your convenience. If you can get DimOS running on temple OS with a package manager you wrote yourself, all the power to you. --- From afd5a96eccfdf2b28cc9cbcafa08cf413805e68c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:07:42 -0800 Subject: [PATCH 082/136] Update README.md --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index 832275ccbd..1042b0a41b 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -273,7 +273,7 @@ You can enable a tag by selecting -m - these are configured in `./pyp - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large file management](/docs/data.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Did we mention smaller PR's are better? Smaller PR's are better. From 217e880b49193166e5ab1edbaf1ddcacf82afffa Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:11:59 -0800 Subject: [PATCH 083/136] Update README.md --- README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index b45451dc9b..3c9f65da49 100644 --- a/README.md +++ b/README.md @@ -189,24 +189,25 @@ if __name__ == "__main__": - Rpc methods get called using threads, meaning two rpc methods can be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. - The start/stop methods always need to be an rpc because they are called externally. +### Monitoring & Debugging +In addition to rerun logging, DimOS comes with a number of monitoring tools: +- Run `lcmspy` to see how fast messages are being published on streams. +- Run `skillspy` to see how skills are being called, how long they are running, which are active, etc. +- Run `agentspy` to see the agent's status over time. +- If you suspect there is a bug within DimOS itself, you can enable extreme logging by prefixing the dimos command with `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 `. Ex: `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos --replay run unitree-go2` -# How does DimOS work? +# How does Dimensional work? Concepts: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. - [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. - [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). -- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): Pretty much an RPC, call but it can be called by an AI agent (they're tools for an AI). +- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. -There are also many monitoring tools: -- Run `lcmspy` to see how fast messages are being published on streams -- Run `skillspy` to see how skills are being called, how long they are running, which are active, etc -- Run `agentspy` to see the agent's status over time - ## Contributing / Building From Source For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. From e3329b8b594fd771867f08947cee41743b433346 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:22:36 -0800 Subject: [PATCH 084/136] switch to dev branch for development --- docs/development/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index 1042b0a41b..e3e572a776 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -43,7 +43,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main git@github.com:dimensionalOS/dimos.git +git clone -b dev git@github.com:dimensionalOS/dimos.git cd dimos @@ -155,7 +155,7 @@ mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main git@github.com:dimensionalOS/dimos.git +git clone -b dev git@github.com:dimensionalOS/dimos.git cd dimos # activate the nix .envrc @@ -194,7 +194,7 @@ mkdir -p "$HOME/.config/nix"; echo "experimental-features = nix-command flakes" # this allows getting large files on-demand export GIT_LFS_SKIP_SMUDGE=1 -git clone -b main git@github.com:dimensionalOS/dimos.git +git clone -b dev git@github.com:dimensionalOS/dimos.git cd dimos # activate the nix development shell From 59d672d288844e7a54ff95e492d19b3db70d72c5 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:42:45 -0800 Subject: [PATCH 085/136] fixup linked files --- README.md | 6 +++--- .../hardware/manipulators/{manipulation.md => README.md} | 0 docs/development/README.md | 8 ++++---- 3 files changed, 7 insertions(+), 7 deletions(-) rename dimos/hardware/manipulators/{manipulation.md => README.md} (100%) diff --git a/README.md b/README.md index 3c9f65da49..1383e71b45 100644 --- a/README.md +++ b/README.md @@ -203,9 +203,9 @@ In addition to rerun logging, DimOS comes with a number of monitoring tools: Concepts: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. - [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. -- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other. -- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). -- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). +- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. +- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). +- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. ## Contributing / Building From Source diff --git a/dimos/hardware/manipulators/manipulation.md b/dimos/hardware/manipulators/README.md similarity index 100% rename from dimos/hardware/manipulators/manipulation.md rename to dimos/hardware/manipulators/README.md diff --git a/docs/development/README.md b/docs/development/README.md index 418c787922..534be7693e 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -229,8 +229,8 @@ This will save the rerun data to `rerun.json` in the current directory. ## Where is `` located? (Architecture) * If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) -* If you want to add a camera driver see [depth_camera_integration.md](/docs/depth_camera_integration.md) -* For edits to manipulation see [manipulation.md](/docs/development/manipulation.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) +* If you want to add a camera driver see [depth_camera_integration.md](/docs/development/depth_camera_integration.md) +* For edits to manipulation see [manipulation](/dimos/hardware/manipulators/REAME.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. @@ -267,11 +267,11 @@ You can enable a tag by selecting -m - these are configured in `./pyp - Open the PR against the `dev` branch (not `main`). - **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. -- If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. +- If you're writing documentation, see [writing docs](/docs/development/writing_docs.md) - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large file management](/docs/data.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Did we mention smaller PR's are better? Smaller PR's are better. From 916dd0611b96b796855b672b7d0c1fbdeb6a1e30 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:43:07 -0800 Subject: [PATCH 086/136] add blueprints back --- docs/concepts/blueprints.md | 319 ++++++++++++++++++++++++++++++++++++ 1 file changed, 319 insertions(+) create mode 100644 docs/concepts/blueprints.md diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md new file mode 100644 index 0000000000..0a3e2ceaf5 --- /dev/null +++ b/docs/concepts/blueprints.md @@ -0,0 +1,319 @@ +# Blueprints + +Blueprints (`ModuleBlueprint`) are instructions for how to initialize a `Module`. + +You don't typically want to run a single module, so multiple blueprints are handled together in `ModuleBlueprintSet`. + +You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: + +```python +blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') +``` + +But the same thing can be acomplished more succinctly as: + +```python +connection = ConnectionModule.blueprint +``` + +Now you can create the blueprint with: + +```python +blueprint = connection('arg1', 'arg2', kwarg='value') +``` + +## Linking blueprints + +You can link multiple blueprints together with `autoconnect`: + +```python +blueprint = autoconnect( + module1(), + module2(), + module3(), +) +``` + +`blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: + +```python +expanded_blueprint = autoconnect( + blueprint, + module4(), + module5(), +) +``` + +Blueprints are frozen data classes, and `autoconnect()` always constructs an expanded blueprint so you never have to worry about changes in one affecting the other. + +### Duplicate module handling + +If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: + +```python +blueprint = autoconnect( + module_a(arg1=1), + module_b(), + module_a(arg1=2), # This one is used, the first is discarded +) +``` + +This is so you can "inherit" from one blueprint but override something you need to change. + +## How transports are linked + +Imagine you have this code: + +```python +class ModuleA(Module): + image: Out[Image] + start_explore: Out[Bool] + +class ModuleB(Module): + image: In[Image] + begin_explore: In[Bool] + +module_a = partial(create_module_blueprint, ModuleA) +module_b = partial(create_module_blueprint, ModuleB) + +autoconnect(module_a(), module_b()) +``` + +Connections are linked based on `(property_name, object_type)`. In this case `('image', Image)` will be connected between the two modules, but `begin_explore` will not be linked to `start_explore`. + +## Topic names + +By default, the name of the property is used to generate the topic name. So for `image`, the topic will be `/image`. + +The property name is used only if it's unique. If two modules have the same property name with different types, then both get a random topic such as `/SGVsbG8sIFdvcmxkI`. + +If you don't like the name you can always override it like in the next section. + +## Which transport is used? + +By default `LCMTransport` is used if the object supports `lcm_encode`. If it doesn't `pLCMTransport` is used (meaning "pickled LCM"). + +You can override transports with the `transports` method. It returns a new blueprint in which the override is set. + +```python +blueprint = autoconnect(...) +expanded_blueprint = autoconnect(blueprint, ...) +blueprint = blueprint.transports({ + ("image", Image): pSHMTransport( + "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE + ), + ("start_explore", Bool): pLCMTransport(), +}) +``` + +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. + +## Remapping connections + +Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: + +```python +class ConnectionModule(Module): + color_image: Out[Image] # Outputs on 'color_image' + +class ProcessingModule(Module): + rgb_image: In[Image] # Expects input on 'rgb_image' + +# Without remapping, these wouldn't connect automatically +# With remapping, color_image is renamed to rgb_image +blueprint = ( + autoconnect( + ConnectionModule.blueprint(), + ProcessingModule.blueprint(), + ) + .remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), + ]) +) +``` + +After remapping: +- The `color_image` output from `ConnectionModule` is treated as `rgb_image` +- It automatically connects to any module with an `rgb_image` input of type `Image` +- The topic name becomes `/rgb_image` instead of `/color_image` + +If you want to override the topic, you still have to do it manually: + +```python +blueprint +.remappings([ + (ConnectionModule, 'color_image', 'rgb_image'), +]) +.transports({ + ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), +}) +``` + +## Overriding global configuration. + +Each module can optionally take a `global_config` option in `__init__`. E.g.: + +```python +class ModuleA(Module): + + def __init__(self, global_config: GlobalConfig | None = None): + ... +``` + +The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: + +```python +blueprint = blueprint.global_config(n_dask_workers=8) +``` + +## Calling the methods of other modules + +Imagine you have this code: + +```python +class ModuleA(Module): + + @rpc + def get_time(self) -> str: + ... + +class ModuleB(Module): + def request_the_time(self) -> None: + ... +``` + +And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. + +To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. + +```python +class ModuleB(Module): + rpc_calls: list[str] = [ + "ModuleA.get_time", + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("ModuleA.get_time") + print(get_time_rpc()) +``` + +You can also request multiple methods at a time: + +```python +method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +``` + +## Alternative RPC calls + +There is an alternative way of receiving RPC methods. It is useful when you want to perform an action at the time you receive the RPC methods. + +You can use it by defining a method like `set__`: + +```python +class ModuleB(Module): + @rpc # Note that it has to be an rpc method. + def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: + self._get_time = rpc_call + self._get_time.set_rpc(self.rpc) + + def request_the_time(self) -> None: + print(self._get_time()) +``` + +Note that `RpcCall.rpc` does not serialize, so you have to set it to the one from the module with `rpc_call.set_rpc(self.rpc)` + +## Calling an interface + +In the previous examples, you can only call methods in a module called `ModuleA`. But what if you want to deploy an alternative module in your blueprint? + +You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. + +```python +class TimeInterface(ABC): + @abstractmethod + def get_time(self): ... + +class ProperTime(TimeInterface): + def get_time(self): + return "13:00" + +class BadTime(TimeInterface): + def get_time(self): + return "01:00 PM" + + +class ModuleB(Module): + rpc_calls: list[str] = [ + "TimeInterface.get_time", # TimeInterface instead of ProperTime or BadTime + ] + + def request_the_time(self) -> None: + get_time_rpc = self.get_rpc_calls("TimeInterface.get_time") + print(get_time_rpc()) +``` + +The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: + +```python +blueprint = autoconnect( + ProperTime.blueprint(), + # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time + ModuleB.blueprint(), +) +``` + +If both are deployed, the blueprint will throw an error because it's ambiguous. + +## Defining skills + +Skills have to be registered with `AgentSpec.register_skills(self)`. + +```python +class SomeSkill(Module): + + @skill + def some_skill(self) -> None: + ... + + @rpc + def set_AgentSpec_register_skills(self, register_skills: RpcCall) -> None: + register_skills.set_rpc(self.rpc) + register_skills(RPCClient(self, self.__class__)) + + # The agent is just interested in the `@skill` methods, so you'll need this if your class + # has things that cannot be pickled. + def __getstate__(self): + pass + def __setstate__(self, _state): + pass +``` + +Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: + +```python +class SomeSkill(SkillModule): + + @skill + def some_skill(self) -> None: + ... +``` + +## Building + +All you have to do to build a blueprint is call: + +```python +module_coordinator = blueprint.build(global_config=config) +``` + +This returns a `ModuleCoordinator` instance that manages all deployed modules. + +### Running and shutting down + +You can block the thread until it exits with: + +```python +module_coordinator.loop() +``` + +This will wait for Ctrl+C and then automatically stop all modules and clean up resources. From 14952593f819fbe4f62ee5fc9361c5801818046c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:43:57 -0800 Subject: [PATCH 087/136] fix typo --- docs/concepts/blueprints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 0a3e2ceaf5..7ddd75a6d8 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -10,7 +10,7 @@ You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') ``` -But the same thing can be acomplished more succinctly as: +But the same thing can be accomplished more succinctly as: ```python connection = ConnectionModule.blueprint From 1d72e650d4ba2a740180c838ff6344aa59e8e4a1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 14:48:18 -0800 Subject: [PATCH 088/136] get Paul's blueprints docs closer to runnable by adding imports --- docs/concepts/blueprints.md | 41 +++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 7ddd75a6d8..6caa1f3004 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -7,6 +7,18 @@ You don't typically want to run a single module, so multiple blueprints are hand You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: ```python +from dimos.core.blueprints import create_module_blueprint +from dimos.core import Module, rpc + +class ConnectionModule(Module): + @rpc + def start(): + pass + + @rpc + def stop(): + pass + blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') ``` @@ -27,6 +39,8 @@ blueprint = connection('arg1', 'arg2', kwarg='value') You can link multiple blueprints together with `autoconnect`: ```python +from dimos.core.blueprints import autoconnect + blueprint = autoconnect( module1(), module2(), @@ -65,6 +79,11 @@ This is so you can "inherit" from one blueprint but override something you need Imagine you have this code: ```python +from functools import partial + +from dimos.core.blueprints import create_module_blueprint, autoconnect +from dimos.core import Module, rpc + class ModuleA(Module): image: Out[Image] start_explore: Out[Bool] @@ -113,6 +132,9 @@ Note: `expanded_blueprint` does not get the transport overrides because it's cre Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: ```python +from dimos.core.blueprints import autoconnect +from dimos.core import Module, rpc + class ConnectionModule(Module): color_image: Out[Image] # Outputs on 'color_image' @@ -154,6 +176,8 @@ blueprint Each module can optionally take a `global_config` option in `__init__`. E.g.: ```python +from dimos.core import Module, rpc + class ModuleA(Module): def __init__(self, global_config: GlobalConfig | None = None): @@ -171,6 +195,8 @@ blueprint = blueprint.global_config(n_dask_workers=8) Imagine you have this code: ```python +from dimos.core import Module, rpc + class ModuleA(Module): @rpc @@ -187,6 +213,8 @@ And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. ```python +from dimos.core import Module, rpc + class ModuleB(Module): rpc_calls: list[str] = [ "ModuleA.get_time", @@ -210,6 +238,8 @@ There is an alternative way of receiving RPC methods. It is useful when you want You can use it by defining a method like `set__`: ```python +from dimos.core import Module, rpc + class ModuleB(Module): @rpc # Note that it has to be an rpc method. def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: @@ -229,6 +259,10 @@ In the previous examples, you can only call methods in a module called `ModuleA` You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. ```python +from abc import ABC, abstractmethod + +from dimos.core import Module, rpc + class TimeInterface(ABC): @abstractmethod def get_time(self): ... @@ -269,6 +303,10 @@ If both are deployed, the blueprint will throw an error because it's ambiguous. Skills have to be registered with `AgentSpec.register_skills(self)`. ```python +from dimos.core import Module, rpc +from dimos.core.skill_module import SkillModule +from dimos.protocol.skill.skill import skill + class SomeSkill(Module): @skill @@ -291,6 +329,9 @@ class SomeSkill(Module): Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: ```python +from dimos.core.skill_module import SkillModule +from dimos.protocol.skill.skill import skill + class SomeSkill(SkillModule): @skill From dccc899647aff0397e4722d69d4129325cd27237 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 16:10:17 -0800 Subject: [PATCH 089/136] add rpc call and skill examples --- README.md | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/README.md b/README.md index 1383e71b45..f4c7063d93 100644 --- a/README.md +++ b/README.md @@ -167,6 +167,116 @@ if __name__ == "__main__": ).build().loop() ``` +#### RPC calls between modules + +Call another module's `@rpc` method by listing it in `rpc_calls` and grabbing it with `get_rpc_calls`. + +```py +from dimos.core import Module, rpc +from dimos.core.blueprints import autoconnect + + +class Counter(Module): + def __init__(self) -> None: + super().__init__() + self._count = 0 + + @rpc + def start(self) -> None: + super().start() + print("Counter ready") + + @rpc + def add(self, amount: int = 1) -> int: + """Add to the running total and return the new count.""" + self._count += amount + print(f"Counter now {self._count}") + return self._count + + @rpc + def stop(self) -> None: + super().stop() + + +class Client(Module): + # ask for the Counter.add RPC, then call it from start() + rpc_calls = ["Counter.add"] + + @rpc + def start(self) -> None: + super().start() + counter_add = self.get_rpc_calls("Counter.add") + counter_add(2) + final_value = counter_add(3) + print(f"Remote counter ended at {final_value}") + + @rpc + def stop(self) -> None: + super().stop() + + +if __name__ == "__main__": + autoconnect( + Counter.blueprint(), + Client.blueprint(), + ).build().loop() +``` + +#### Skills in a module + +Define skills with `@skill`, register them with a `SkillCoordinator`, and retrieve the results as they stream back. + +```py +import asyncio + +from dimos.core import Module, rpc +from dimos.protocol.skill.coordinator import SkillCoordinator +from dimos.protocol.skill.skill import skill +from dimos.protocol.skill.type import Reducer, Stream + + +class Calculator(Module): + @rpc + def start(self) -> None: + super().start() + + @rpc + def stop(self) -> None: + super().stop() + + @skill() + def add(self, x: int, y: int) -> int: + return x + y + + @skill(stream=Stream.passive, reducer=Reducer.latest) # type: ignore[arg-type] + def count_to(self, n: int): + for i in range(1, n + 1): + yield i + + +async def main() -> None: + skills = Calculator() + coordinator = SkillCoordinator() + coordinator.register_skills(skills) + coordinator.start() + + coordinator.call_skill("add-1", "add", {"args": [2, 3]}) + coordinator.call_skill("count-1", "count_to", {"args": [4]}) + + while await coordinator.wait_for_updates(timeout=1): + updates = coordinator.generate_snapshot(clear=True) + for call_id, state in updates.items(): + print(f"{call_id} -> {state.content()}") + if not coordinator.has_passive_skills(): + break + + coordinator.stop() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + #### Note: Many More Examples in the [Examples Folder](./examples) ### How do custom modules work? (Example breakdown) From e07908b31dbb2de7de68dbbdfb6eba0b70817b6a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 16:19:20 -0800 Subject: [PATCH 090/136] better docs for skills --- README.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f4c7063d93..db3f1ac6a7 100644 --- a/README.md +++ b/README.md @@ -296,8 +296,23 @@ if __name__ == "__main__": - Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())` - What about `@rpc`? - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. - - Rpc methods get called using threads, meaning two rpc methods can be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. - The start/stop methods always need to be an rpc because they are called externally. + - Under the hood rpc methods get called using threads. This means it is possible for two rpc methods to be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. + - To call another module's RPC, add its name to `rpc_calls` (ex: `["Counter.add"]`), then `self.get_rpc_calls("Counter.add")` returns a callable that forwards across transports. The method key is always `ClassName.method_name`. Note: this API will likely be replaced with a more convenient API. +- What about skills? + - Skills are upgraded rpc methods designed to be called by an AI agent. + - Because skills can be dynamically added/removed, they are registered with a SkillCoordinator. + +### Documentation & Concepts + +If you you need more information on how DimOS works, check out the following links: + +- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. +- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. +- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). +- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. ### Monitoring & Debugging From aa7223329d90f4faf0785548c165e5a87e57e82f Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 16:21:14 -0800 Subject: [PATCH 091/136] improve contributing --- README.md | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index db3f1ac6a7..0034d9fa31 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,8 @@

Key Features • How To Use • - Contributing • License + Contributing • + License

@@ -102,6 +103,10 @@ source .venv/bin/activate humancli ``` +## Contributing / Building From Source + +We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. + # How do I use it as a library? ### Simple Camera Activation @@ -322,23 +327,6 @@ In addition to rerun logging, DimOS comes with a number of monitoring tools: - Run `agentspy` to see the agent's status over time. - If you suspect there is a bug within DimOS itself, you can enable extreme logging by prefixing the dimos command with `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 `. Ex: `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos --replay run unitree-go2` - -# How does Dimensional work? - -Concepts: -- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. -- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. -- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. -- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). -- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). -- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. - -## Contributing / Building From Source - -For development, we optimize for flexibility—whether you love Docker, Nix, or have nothing but **notepad.exe** and a dream, you’re good to go. Open up the [Development Guide](/docs/development/README.md) to see the extra steps for setting up development environments. - -We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. - # License DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. From a1c9ac0652a630f41f6b0a7b7e37d6083fccaecd Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Thu, 15 Jan 2026 18:07:12 -0800 Subject: [PATCH 092/136] changes for paul --- README.md | 8 +++++--- docs/development/README.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3c9f65da49..748391e557 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin # OPTION 1: install dimos in a virtualenv uv venv && . .venv/bin/activate uv pip install 'dimos[base,unitree]' +# replay recorded data to test that the system is working dimos --replay run unitree-go2 # OPTION 2: if you want to test out dimos without installing run: @@ -67,11 +68,12 @@ uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 #### Control a robot in a simulation (no robot required) -```bash +After running the commads below, open http://localhost:7779/command-center to control the robot movement. + +```sh export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors # ignore the warp warnings dimos --simulation run unitree-go2 -# open http://localhost:7779/command-center in your browser to control the robot movement ``` #### Get it working on a physical robot! @@ -151,7 +153,7 @@ class Listener(Module): self.grayscale_image.publish(img.to_grayscale()) unsubscribe_func = self.color_image.subscribe(callback_func) - # disposables will be called when the module is stopped + # the unsubscribe_func be called when the module is stopped self._disposables.add(Disposable( unsubscribe_func )) diff --git a/docs/development/README.md b/docs/development/README.md index e3e572a776..a4eef2ef8d 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -257,7 +257,6 @@ pytest # run all tests at or below the current directory | Filter tests by name | `pytest -k ""` | | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | -| Run all pre-commit hooks | `pre-commit run --all-files` | We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) @@ -270,6 +269,7 @@ You can enable a tag by selecting -m - these are configured in `./pyp - **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. - If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. +- If you get mypy errors, please fix them. Don't just add # type: ignore. Please first understand why mypy is complaining and try to fix it. It's only okay to ignore if the issue cannot be fixed. - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. From 6a10351b2d9cf6ad21045bfe92c4e2e77c235462 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 16 Jan 2026 15:24:35 -0800 Subject: [PATCH 093/136] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 748391e557..7c0ffe06f1 100644 --- a/README.md +++ b/README.md @@ -73,14 +73,14 @@ After running the commads below, open http://localhost:7779/command-center to co ```sh export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors # ignore the warp warnings -dimos --simulation run unitree-go2 +dimos --viewer-backend rerun-web --simulation run unitree-go2 ``` #### Get it working on a physical robot! ```sh export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE -dimos run unitree-go2 +dimos --viewer-backend rerun-web run unitree-go2 ``` #### Have it controlled by AI! @@ -89,7 +89,7 @@ WARNING: This is a demo showing the **connection** between AI and robotic contro ```sh export OPENAI_API_KEY= -dimos run unitree-go2-agentic +dimos --viewer-backend rerun-web run unitree-go2-agentic ``` After running that, open a new terminal and run the following to start giving instructions to the agent. From 5bcaf2af4de9f5f27eb5e8a04c6c33c93f6a4a4a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 16 Jan 2026 16:06:54 -0800 Subject: [PATCH 094/136] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7c0ffe06f1..cf6c5322b3 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,7 @@ from dimos.hardware.sensors.camera.module import CameraModule if __name__ == "__main__": autoconnect( + # technically autoconnect is not needed because we only have 1 module CameraModule.blueprint() ).build().loop() ``` From 653276f0da8e39cfab6a2e9baf11c0bbd8290a7e Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Sat, 17 Jan 2026 15:37:54 -0800 Subject: [PATCH 095/136] fix broken link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cf6c5322b3..589aabb43a 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ [![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square&logo=docker&logoColor=white)](https://www.docker.com/)

- Key Features • + Key Features • How To Use • Contributing • License

From b333d7889d98f99850b1bfcc18a783a1ef6591b0 Mon Sep 17 00:00:00 2001 From: Paul Nechifor Date: Sun, 18 Jan 2026 01:56:14 +0200 Subject: [PATCH 096/136] update broken link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 589aabb43a..c2f9992abd 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ [![Docker](https://img.shields.io/badge/Docker-ready-2496ED?style=flat-square&logo=docker&logoColor=white)](https://www.docker.com/)

- Key Features • + Key Features • How To Use • Contributing • License

From 0eebfb9668271305c81e2bec27c14a105f2d079d Mon Sep 17 00:00:00 2001 From: stash Date: Sun, 18 Jan 2026 19:14:47 -0800 Subject: [PATCH 097/136] Readme changes --- README.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index c2f9992abd..1ae7f13793 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@
banner_bordered_trimmed -

The Open-Source Framework for Robotic Intelligence

+

The Agentive Operating System for Generalist Robotics


@@ -37,28 +37,36 @@ The python library comes with a rich set of integrations; visualizers, spatial r ### Installation -- Linux is supported, with tests being performed on Ubuntu 22.04 and 24.04 -- MacOS support is in beta, you're welcome to try it *but expect inconsistent/flakey behavior (rather than errors/crashing)* - - instead of the apt-get command below run: `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` +Supported/tested matrix: + +| Platform | Status | Tested | Required System deps | +| --- | --- | --- | --- | +| Linux | supported | Ubuntu 22.04, 24.04 | See below | +| macOS | experimental beta | not CI-tested | `brew install gnu-sed gcc portaudio git-lfs libjpeg-turbo python` | + +Note: macOS is usable but expect inconsistent/flaky behavior (rather than hard errors/crashes). ```sh sudo apt-get update sudo apt-get install -y curl g++ portaudio19-dev git-lfs libturbojpeg python3-dev # install uv for python curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" +``` + +Option 1: Install in a virtualenv -# -# NOTE!!! the first time, you're going to have an empty/black rerun window for a while -# -# the command needs to download the replay file (2.4gb), which takes a bit +```sh -# OPTION 1: install dimos in a virtualenv uv venv && . .venv/bin/activate uv pip install 'dimos[base,unitree]' # replay recorded data to test that the system is working +# IMPORTANT: First replay run will show a black rerun window while 2.4 GB downloads from LFS dimos --replay run unitree-go2 +``` -# OPTION 2: if you want to test out dimos without installing run: +Option 2: Run without installing + +```sh uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 ``` From a0226f47e75c50b4c0b669f8337628a80a96bbb0 Mon Sep 17 00:00:00 2001 From: stash Date: Sun, 18 Jan 2026 22:19:11 -0800 Subject: [PATCH 098/136] readme reorg --- README.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 1ae7f13793..6eae8df804 100644 --- a/README.md +++ b/README.md @@ -72,11 +72,10 @@ uvx --from 'dimos[base,unitree]' dimos --replay run unitree-go2 -### Usage +### Dimensional Usage #### Control a robot in a simulation (no robot required) -After running the commads below, open http://localhost:7779/command-center to control the robot movement. ```sh export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors @@ -84,16 +83,20 @@ export DISPLAY=:1 # Or DISPLAY=:0 if getting GLFW/OpenGL X11 errors dimos --viewer-backend rerun-web --simulation run unitree-go2 ``` -#### Get it working on a physical robot! +#### Control a real robot (Unitree Go2 over WebRTC) ```sh -export ROBOT_IP=PUT_YOUR_IP_ADDR_HERE +export ROBOT_IP= dimos --viewer-backend rerun-web run unitree-go2 ``` -#### Have it controlled by AI! +After running dimOS open http://localhost:7779 to control robot movement. -WARNING: This is a demo showing the **connection** between AI and robotic control -- not a demo of a super-intelligent AI. Be ready to physically prevent your robot from taking dumb physical actions. +#### Dimensional Agents + +> \[!NOTE] +> +> **Experimental Beta: Potential unstoppable robot sentience** ```sh export OPENAI_API_KEY= @@ -105,8 +108,6 @@ After running that, open a new terminal and run the following to start giving in # activate the venv in this new terminal source .venv/bin/activate -# Note: after running the next command, WAIT for the agent to connect -# (this will take a while the first time) # then tell the agent "explore the room" # then tell it to go to something, ex: "go to the door" humancli From c53a8346a437969e793ec73beff6f1ed56b2a116 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 10:47:55 -0800 Subject: [PATCH 099/136] rename so that it renders on github --- docs/api/sensor_streams/{index.md => README.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/api/sensor_streams/{index.md => README.md} (100%) diff --git a/docs/api/sensor_streams/index.md b/docs/api/sensor_streams/README.md similarity index 100% rename from docs/api/sensor_streams/index.md rename to docs/api/sensor_streams/README.md From f932070ee8d682b60a9da84e8ef9f8e3ab485753 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 11:23:27 -0800 Subject: [PATCH 100/136] break up writing documentation --- docs/development/writing_docs.md | 482 ------------------ docs/development/writing_docs/README.md | 17 + docs/development/writing_docs/code_blocks.md | 195 +++++++ .../writing_docs/diagram_practices.md | 140 +++++ docs/development/writing_docs/doclinks.md | 119 +++++ 5 files changed, 471 insertions(+), 482 deletions(-) delete mode 100644 docs/development/writing_docs.md create mode 100644 docs/development/writing_docs/README.md create mode 100644 docs/development/writing_docs/code_blocks.md create mode 100644 docs/development/writing_docs/diagram_practices.md create mode 100644 docs/development/writing_docs/doclinks.md diff --git a/docs/development/writing_docs.md b/docs/development/writing_docs.md deleted file mode 100644 index b9ae0b27df..0000000000 --- a/docs/development/writing_docs.md +++ /dev/null @@ -1,482 +0,0 @@ -# Need-to-know Things - -1. How we resolve file references (linking) -2. How we make of our code blocks executable and test them. -3. How we make diagrams - - -
-
- -# 1. Use Doclinks to Resolve file references - -## Syntax - - -| Pattern | Example | -|-------------|-----------------------------------------------------| -| Code file | `[`service/spec.py`]()` → resolves path | -| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | -| Doc link | `[Configuration](.md)` → resolves to doc | - - -## Usage - -```bash -doclinks docs/guide.md # single file -doclinks docs/ # directory -doclinks --dry-run ... # preview only -``` - -## Full Documentation - -
-Click to see full documentation - -## What it does - -When writing docs, you can use placeholder links like: - - -```markdown -See [`service/spec.py`]() for the implementation. -``` - - -Running `doclinks` resolves these to actual paths: - - -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - - -## Features - - -- **Code file links**: `[`filename.py`]()` resolves to the file's path -- **Symbol line linking**: If another backticked term appears on the same line, it finds that symbol in the file and adds `#L`: - ```markdown - See `Configurable` in [`config.py`]() - → [`config.py`](/path/config.py#L42) - ``` -- **Doc-to-doc links**: `[Modules](.md)` resolves to `modules.md` or `modules/index.md` - -- **Multiple link modes**: absolute, relative, or GitHub URLs -- **Watch mode**: Automatically re-process on file changes -- **Ignore regions**: Skip sections with `` comments - -## Usage - -```bash -# Process a single file -doclinks docs/guide.md - -# Process a directory recursively -doclinks docs/ - -# Relative links (from doc location) -doclinks --link-mode relative docs/ - -# GitHub links -doclinks --link-mode github \ - --github-url https://github.com/org/repo docs/ - -# Dry run (preview changes) -doclinks --dry-run docs/ - -# CI check (exit 1 if changes needed) -doclinks --check docs/ - -# Watch mode (auto-update on changes) -doclinks --watch docs/ -``` - -## Options - -| Option | Description | -|--------------------|-------------------------------------------------| -| `--root PATH` | Repository root (default: auto-detect git root) | -| `--link-mode MODE` | `absolute` (default), `relative`, or `github` | -| `--github-url URL` | Base GitHub URL (required for github mode) | -| `--github-ref REF` | Branch/ref for GitHub links (default: `main`) | -| `--dry-run` | Show changes without modifying files | -| `--check` | Exit with error if changes needed (for CI) | -| `--watch` | Watch for changes and re-process | - -## Link patterns - - -| Pattern | Description | -|----------------------|------------------------------------------------| -| `[`file.py`]()` | Code file reference (empty or any link) | -| `[`path/file.py`]()` | Code file with partial path for disambiguation | -| `[`file.py`](#L42)` | Preserves existing line fragments | -| `[Doc Name](.md)` | Doc-to-doc link (resolves by name) | - - -## How resolution works - -The tool builds an index of all files in the repo. For `/dimos/protocol/service/spec.py`, it creates lookup entries for: - -- `spec.py` -- `service/spec.py` -- `protocol/service/spec.py` -- `dimos/protocol/service/spec.py` - -Use longer paths when multiple files share the same name. - -
- - - -
-
- -# 2. Code Blocks Must Be Executable - -We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. - -## Golden Rule - -**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. - -## Installation - -
-Click to see full installation instructions - -### Nix (recommended) - -```sh skip -# (assuming you have nix) - -# Run directly from GitHub -nix run github:leshy/md-babel-py -- run README.md --stdout - -# run locally -nix run . -- run README.md --stdout -``` - -### Docker - -```sh skip -# Pull from Docker Hub -docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout - -# Or build locally via Nix -nix build .#docker # builds tarball to ./result -docker load < result # loads image from tarball -docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout -``` - -### pipx - -```sh skip -pipx install md-babel-py -# or: uv pip install md-babel-py -md-babel-py run README.md --stdout -``` - -If not using nix or docker, evaluators require system dependencies: - -| Language | System packages | -|-----------|-----------------------------| -| python | python3 | -| node | nodejs | -| dot | graphviz | -| asymptote | asymptote, texlive, dvisvgm | -| pikchr | pikchr | -| openscad | openscad, xvfb, imagemagick | -| diagon | diagon | - -```sh skip -# Arch Linux -sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick - -# Debian/Ubuntu -sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad -``` - -Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. - -## Usage - -```sh skip -# Edit file in-place -md-babel-py run document.md - -# Output to separate file -md-babel-py run document.md --output result.md - -# Print to stdout -md-babel-py run document.md --stdout - -# Only run specific languages -md-babel-py run document.md --lang python,sh - -# Dry run - show what would execute -md-babel-py run document.md --dry-run -``` - -
- - -## Running - -```sh skip -md-babel-py run document.md # edit in-place -md-babel-py run document.md --stdout # preview to stdout -md-babel-py run document.md --dry-run # show what would run -``` - -## Supported Languages - -Python, Shell (sh), Node.js, plus visualization: Matplotlib, Graphviz, Pikchr, Asymptote, OpenSCAD, Diagon. - -## Code Block Flags - -Add flags after the language identifier: - -| Flag | Effect | -|------|--------| -| `session=NAME` | Share state between blocks with same session name | -| `output=path.png` | Write output to file instead of inline | -| `no-result` | Execute but don't insert result | -| `skip` | Don't execute this block | -| `expected-error` | Block is expected to fail | - -## Examples - -# md-babel-py - -Execute code blocks in markdown files and insert the results. - -![Demo](assets/screencast.gif) - -**Use cases:** -- Keep documentation examples up-to-date automatically -- Validate code snippets in docs actually work -- Generate diagrams and charts from code in markdown -- Literate programming with executable documentation - -## Languages - -### Shell - -```sh -echo "cwd: $(pwd)" -``` - - -``` -cwd: /work -``` - -### Python - -```python session=example -a = "hello world" -print(a) -``` - - -``` -hello world -``` - -Sessions preserve state between code blocks: - -```python session=example -print(a, "again") -``` - - -``` -hello world again -``` - -### Node.js - -```node -console.log("Hello from Node.js"); -console.log(`Node version: ${process.version}`); -``` - - -``` -Hello from Node.js -Node version: v22.21.1 -``` - -### Matplotlib - -```python output=assets/matplotlib-demo.svg -import matplotlib.pyplot as plt -import numpy as np -plt.style.use('dark_background') -x = np.linspace(0, 4 * np.pi, 200) -plt.figure(figsize=(8, 4)) -plt.plot(x, np.sin(x), label='sin(x)', linewidth=2) -plt.plot(x, np.cos(x), label='cos(x)', linewidth=2) -plt.xlabel('x') -plt.ylabel('y') -plt.legend() -plt.grid(alpha=0.3) -plt.savefig('{output}', transparent=True) -``` - - -![output](assets/matplotlib-demo.svg) - - - - - - -
-
- -# 3. Diagrams - -We have many diagraming tools. View source code of this page to see examples. - - -### Pikchr - -SQLite's diagram language: - -
-diagram source - -```pikchr fold output=assets/pikchr-demo.svg -color = white -fill = none -linewid = 0.4in - -# Input file -In: file "README.md" fit -arrow - -# Processing -Parse: box "Parse" rad 5px fit -arrow -Exec: box "Execute" rad 5px fit - -# Fan out to languages -arrow from Exec.e right 0.3in then up 0.4in then right 0.3in -Sh: oval "Shell" fit -arrow from Exec.e right 0.3in then right 0.3in -Node: oval "Node" fit -arrow from Exec.e right 0.3in then down 0.4in then right 0.3in -Py: oval "Python" fit - -# Merge back -X: dot at (Py.e.x + 0.3in, Node.e.y) invisible -line from Sh.e right until even with X then down to X -line from Node.e to X -line from Py.e right until even with X then up to X -Out: file "README.md" fit with .w at (X.x + 0.3in, X.y) -arrow from X to Out.w -``` - -
- - -![output](assets/pikchr-demo.svg) - -### Asymptote - -Vector graphics: - -```asymptote output=assets/histogram.svg -import graph; -import stats; - -size(400,200,IgnoreAspect); -defaultpen(white); - -int n=10000; -real[] a=new real[n]; -for(int i=0; i < n; ++i) a[i]=Gaussrand(); - -draw(graph(Gaussian,min(a),max(a)),orange); - -int N=bins(a); - -histogram(a,min(a),max(a),N,normalize=true,low=0,rgb(0.4,0.6,0.8),rgb(0.2,0.4,0.6),bars=true); - -xaxis("$x$",BottomTop,LeftTicks,p=white); -yaxis("$dP/dx$",LeftRight,RightTicks(trailingzero),p=white); -``` - - -![output](assets/histogram.svg) - -### Graphviz - -```dot output=assets/graph.svg -A -> B -> C -A -> C -``` - - -![output](assets/graph.svg) - -### OpenSCAD - -```openscad output=assets/cube-sphere.png -cube([10, 10, 10]); -sphere(r=7); -``` - - -![output](assets/cube-sphere.png) - -### Diagon - -ASCII art diagrams: - -```diagon mode=Math -1 + 1/2 + sum(i,0,10) -``` - - -``` - 10 - ___ - 1 ╲ -1 + ─ + ╱ i - 2 ‾‾‾ - 0 -``` - -```diagon mode=GraphDAG -A -> B -> C -A -> C -``` - - -``` -┌───┐ -│A │ -└┬─┬┘ - │┌▽┐ - ││B│ - │└┬┘ -┌▽─▽┐ -│C │ -└───┘ -``` - -## Reference - -| Element | Syntax | -|---------|--------| -| Box | `box "text" rad 5px wid Xin ht Yin` | -| Arrow | `arrow right 0.3in` | -| Oval | `oval "text" wid Xin ht Yin` | -| Text | `text "label" at (X, Y)` | -| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | - -See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/development/writing_docs/README.md b/docs/development/writing_docs/README.md new file mode 100644 index 0000000000..a848497e23 --- /dev/null +++ b/docs/development/writing_docs/README.md @@ -0,0 +1,17 @@ +# Writing Docs + +Note: as of the DimOS beta, not all existing docs conform to this guide, but newly added docs should. + +## Need-to-know Things + +1. Where to put your docs. + - Some docs are under `docs/` (like this one) but others are stored in the actual codebase, like `dimos/robots/drones/README.md`. + - If your docs have code examples, and are somewhere under `docs/` those code examples must be executable. See [codeblocks guide](/docs/development/writing_docs/code_blocks.md) for details and instructions on how to execute your code examples. + - If your docs nicely *introduce* a new API, or they are a tutorial, then put them in `docs/concepts/` (even if they are about a specific API). + - If the docs are highly technical or exhaustive there are a three options: + - If your docs are about a user-facing API (ex: the reader can follow your instructions without cloning dimos) then put them in `docs/api/`. + - Otherwise (if the reader is modifying their own copy of the dimos codebase) then your docs have two options: + 1. You can choose to store your docs next to relevant python files (ex: `dimos/robots/drones/README.md`), and we are less strict about the contents (code examples don't need to be executable) **BUT**, you need edit something in `docs/development/` or `docs/api/` that to add a reference/link to those docs (don't create "dangling" documentation). + 2. Alternatively, you can put your docs in `docs/development/`. Code examples there should be executable. +2. Even if you know how to link to other docs, read our [how we do doc linking guide](/docs/development/writing_docs/doclinks.md). +3. Even if you know how to create diagrams on your own, read our [how we do diagrams guide](/docs/development/writing_docs/diagram_practices.md). diff --git a/docs/development/writing_docs/code_blocks.md b/docs/development/writing_docs/code_blocks.md new file mode 100644 index 0000000000..5b5b8a4dfc --- /dev/null +++ b/docs/development/writing_docs/code_blocks.md @@ -0,0 +1,195 @@ +# Code Blocks Must Be Executable + +We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. + +## Golden Rule + +**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. + +## Installation + +
+Click to see full installation instructions + +### Nix (recommended) + +```sh skip +# (assuming you have nix) + +# Run directly from GitHub +nix run github:leshy/md-babel-py -- run README.md --stdout + +# run locally +nix run . -- run README.md --stdout +``` + +### Docker + +```sh skip +# Pull from Docker Hub +docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout + +# Or build locally via Nix +nix build .#docker # builds tarball to ./result +docker load < result # loads image from tarball +docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout +``` + +### pipx + +```sh skip +pipx install md-babel-py +# or: uv pip install md-babel-py +md-babel-py run README.md --stdout +``` + +If not using nix or docker, evaluators require system dependencies: + +| Language | System packages | +|-----------|-----------------------------| +| python | python3 | +| node | nodejs | +| dot | graphviz | +| asymptote | asymptote, texlive, dvisvgm | +| pikchr | pikchr | +| openscad | openscad, xvfb, imagemagick | +| diagon | diagon | + +```sh skip +# Arch Linux +sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick + +# Debian/Ubuntu +sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad +``` + +Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. + +## Usage + +```sh skip +# Edit file in-place +md-babel-py run document.md + +# Output to separate file +md-babel-py run document.md --output result.md + +# Print to stdout +md-babel-py run document.md --stdout + +# Only run specific languages +md-babel-py run document.md --lang python,sh + +# Dry run - show what would execute +md-babel-py run document.md --dry-run +``` + +
+ + +## Running + +```sh skip +md-babel-py run document.md # edit in-place +md-babel-py run document.md --stdout # preview to stdout +md-babel-py run document.md --dry-run # show what would run +``` + +## Supported Languages + +Python, Shell (sh), Node.js, plus visualization: Matplotlib, Graphviz, Pikchr, Asymptote, OpenSCAD, Diagon. + +## Code Block Flags + +Add flags after the language identifier: + +| Flag | Effect | +|------|--------| +| `session=NAME` | Share state between blocks with same session name | +| `output=path.png` | Write output to file instead of inline | +| `no-result` | Execute but don't insert result | +| `skip` | Don't execute this block | +| `expected-error` | Block is expected to fail | + +## Examples + +# md-babel-py + +Execute code blocks in markdown files and insert the results. + +![Demo](assets/screencast.gif) + +**Use cases:** +- Keep documentation examples up-to-date automatically +- Validate code snippets in docs actually work +- Generate diagrams and charts from code in markdown +- Literate programming with executable documentation + +## Languages + +### Shell + +```sh +echo "cwd: $(pwd)" +``` + + +``` +cwd: /work +``` + +### Python + +```python session=example +a = "hello world" +print(a) +``` + + +``` +hello world +``` + +Sessions preserve state between code blocks: + +```python session=example +print(a, "again") +``` + + +``` +hello world again +``` + +### Node.js + +```node +console.log("Hello from Node.js"); +console.log(`Node version: ${process.version}`); +``` + + +``` +Hello from Node.js +Node version: v22.21.1 +``` + +### Matplotlib + +```python output=assets/matplotlib-demo.svg +import matplotlib.pyplot as plt +import numpy as np +plt.style.use('dark_background') +x = np.linspace(0, 4 * np.pi, 200) +plt.figure(figsize=(8, 4)) +plt.plot(x, np.sin(x), label='sin(x)', linewidth=2) +plt.plot(x, np.cos(x), label='cos(x)', linewidth=2) +plt.xlabel('x') +plt.ylabel('y') +plt.legend() +plt.grid(alpha=0.3) +plt.savefig('{output}', transparent=True) +``` + + +![output](assets/matplotlib-demo.svg) diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md new file mode 100644 index 0000000000..7a27d89b9b --- /dev/null +++ b/docs/development/writing_docs/diagram_practices.md @@ -0,0 +1,140 @@ +We have many diagraming tools. View source code of this page to see examples. + +### Pikchr + +SQLite's diagram language: + +
+diagram source + +```pikchr fold output=assets/pikchr-demo.svg +color = white +fill = none +linewid = 0.4in + +# Input file +In: file "README.md" fit +arrow + +# Processing +Parse: box "Parse" rad 5px fit +arrow +Exec: box "Execute" rad 5px fit + +# Fan out to languages +arrow from Exec.e right 0.3in then up 0.4in then right 0.3in +Sh: oval "Shell" fit +arrow from Exec.e right 0.3in then right 0.3in +Node: oval "Node" fit +arrow from Exec.e right 0.3in then down 0.4in then right 0.3in +Py: oval "Python" fit + +# Merge back +X: dot at (Py.e.x + 0.3in, Node.e.y) invisible +line from Sh.e right until even with X then down to X +line from Node.e to X +line from Py.e right until even with X then up to X +Out: file "README.md" fit with .w at (X.x + 0.3in, X.y) +arrow from X to Out.w +``` + +
+ + +![output](assets/pikchr-demo.svg) + +### Asymptote + +Vector graphics: + +```asymptote output=assets/histogram.svg +import graph; +import stats; + +size(400,200,IgnoreAspect); +defaultpen(white); + +int n=10000; +real[] a=new real[n]; +for(int i=0; i < n; ++i) a[i]=Gaussrand(); + +draw(graph(Gaussian,min(a),max(a)),orange); + +int N=bins(a); + +histogram(a,min(a),max(a),N,normalize=true,low=0,rgb(0.4,0.6,0.8),rgb(0.2,0.4,0.6),bars=true); + +xaxis("$x$",BottomTop,LeftTicks,p=white); +yaxis("$dP/dx$",LeftRight,RightTicks(trailingzero),p=white); +``` + + +![output](assets/histogram.svg) + +### Graphviz + +```dot output=assets/graph.svg +A -> B -> C +A -> C +``` + + +![output](assets/graph.svg) + +### OpenSCAD + +```openscad output=assets/cube-sphere.png +cube([10, 10, 10]); +sphere(r=7); +``` + + +![output](assets/cube-sphere.png) + +### Diagon + +ASCII art diagrams: + +```diagon mode=Math +1 + 1/2 + sum(i,0,10) +``` + + +``` + 10 + ___ + 1 ╲ +1 + ─ + ╱ i + 2 ‾‾‾ + 0 +``` + +```diagon mode=GraphDAG +A -> B -> C +A -> C +``` + + +``` +┌───┐ +│A │ +└┬─┬┘ + │┌▽┐ + ││B│ + │└┬┘ +┌▽─▽┐ +│C │ +└───┘ +``` + +## Reference + +| Element | Syntax | +|---------|--------| +| Box | `box "text" rad 5px wid Xin ht Yin` | +| Arrow | `arrow right 0.3in` | +| Oval | `oval "text" wid Xin ht Yin` | +| Text | `text "label" at (X, Y)` | +| Named point | `A: box ...` then reference `A.e`, `A.n`, `A.x`, `A.y` | + +See [pikchr.org/home/doc/trunk/doc/userman.md](https://pikchr.org/home/doc/trunk/doc/userman.md) for full documentation. diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md new file mode 100644 index 0000000000..e1cd9d0457 --- /dev/null +++ b/docs/development/writing_docs/doclinks.md @@ -0,0 +1,119 @@ +# Use Doclinks to Resolve file references + +## Syntax + + +| Pattern | Example | +|-------------|-----------------------------------------------------| +| Code file | `[`service/spec.py`]()` → resolves path | +| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | +| Doc link | `[Configuration](.md)` → resolves to doc | + + +## Usage + +```bash +doclinks docs/guide.md # single file +doclinks docs/ # directory +doclinks --dry-run ... # preview only +``` + +## Full Documentation + +
+Click to see full documentation + +## What it does + +When writing docs, you can use placeholder links like: + + +```markdown +See [`service/spec.py`]() for the implementation. +``` + + +Running `doclinks` resolves these to actual paths: + + +```markdown +See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. +``` + + +## Features + + +- **Code file links**: `[`filename.py`]()` resolves to the file's path +- **Symbol line linking**: If another backticked term appears on the same line, it finds that symbol in the file and adds `#L`: + ```markdown + See `Configurable` in [`config.py`]() + → [`config.py`](/path/config.py#L42) + ``` +- **Doc-to-doc links**: `[Modules](.md)` resolves to `modules.md` or `modules/index.md` + +- **Multiple link modes**: absolute, relative, or GitHub URLs +- **Watch mode**: Automatically re-process on file changes +- **Ignore regions**: Skip sections with `` comments + +## Usage + +```bash +# Process a single file +doclinks docs/guide.md + +# Process a directory recursively +doclinks docs/ + +# Relative links (from doc location) +doclinks --link-mode relative docs/ + +# GitHub links +doclinks --link-mode github \ + --github-url https://github.com/org/repo docs/ + +# Dry run (preview changes) +doclinks --dry-run docs/ + +# CI check (exit 1 if changes needed) +doclinks --check docs/ + +# Watch mode (auto-update on changes) +doclinks --watch docs/ +``` + +## Options + +| Option | Description | +|--------------------|-------------------------------------------------| +| `--root PATH` | Repository root (default: auto-detect git root) | +| `--link-mode MODE` | `absolute` (default), `relative`, or `github` | +| `--github-url URL` | Base GitHub URL (required for github mode) | +| `--github-ref REF` | Branch/ref for GitHub links (default: `main`) | +| `--dry-run` | Show changes without modifying files | +| `--check` | Exit with error if changes needed (for CI) | +| `--watch` | Watch for changes and re-process | + +## Link patterns + + +| Pattern | Description | +|----------------------|------------------------------------------------| +| `[`file.py`]()` | Code file reference (empty or any link) | +| `[`path/file.py`]()` | Code file with partial path for disambiguation | +| `[`file.py`](#L42)` | Preserves existing line fragments | +| `[Doc Name](.md)` | Doc-to-doc link (resolves by name) | + + +## How resolution works + +The tool builds an index of all files in the repo. For `/dimos/protocol/service/spec.py`, it creates lookup entries for: + +- `spec.py` +- `service/spec.py` +- `protocol/service/spec.py` +- `dimos/protocol/service/spec.py` + +Use longer paths when multiple files share the same name. + +
From 7ec97472c3f71bd522082c7681bacb46bf71452c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 11:24:06 -0800 Subject: [PATCH 101/136] add minimal API overview --- docs/api/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 docs/api/README.md diff --git a/docs/api/README.md b/docs/api/README.md new file mode 100644 index 0000000000..1a14f83640 --- /dev/null +++ b/docs/api/README.md @@ -0,0 +1,11 @@ +# API + +Note: Please see [Concepts](/docs/concepts/README.md) before diving into these API docs. These docs are designed to be technical, cover specific tooling, and give examples of using those tools (argument types) rather than explaining when to use the tool. + + +# Table of Contents + +- [adding configs to your modules](./configuration.md) +- [controlling how visualizations are rendered](./visualization.md) +- [understanding sensor streams](./sensor_streams/README.md) +- [creating coordinate frames and transforms for your hardware](./transforms.md) From 03ebe5c2b375c9efea326eca733bb1d54b5c7d84 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 11:32:07 -0800 Subject: [PATCH 102/136] add table of contents to concepts --- docs/concepts/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 docs/concepts/README.md diff --git a/docs/concepts/README.md b/docs/concepts/README.md new file mode 100644 index 0000000000..93b25cde5f --- /dev/null +++ b/docs/concepts/README.md @@ -0,0 +1,12 @@ +# Concepts + +If you are trying to learn a new API or know what tools you should use, then you are in the right place. In contrast, if you already know the API you want to use and just need details on how to use it, see [the API reference](/docs/API/README.md). + +## Table of Contents + +- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. +- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. +- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). +- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. From 0e52d06850c5a96d6623629af759f8aebaeeb0de Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 11:37:32 -0800 Subject: [PATCH 103/136] add actual doclinks command so the doclinks docs make sense --- bin/doclinks | 3 +++ docs/development/writing_docs/doclinks.md | 20 ++++++++++---------- 2 files changed, 13 insertions(+), 10 deletions(-) create mode 100755 bin/doclinks diff --git a/bin/doclinks b/bin/doclinks new file mode 100755 index 0000000000..c2b05e8acb --- /dev/null +++ b/bin/doclinks @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +REPO_ROOT=$(git rev-parse --show-toplevel) +python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md index e1cd9d0457..e9a6ee9d8c 100644 --- a/docs/development/writing_docs/doclinks.md +++ b/docs/development/writing_docs/doclinks.md @@ -13,9 +13,9 @@ ## Usage ```bash -doclinks docs/guide.md # single file -doclinks docs/ # directory -doclinks --dry-run ... # preview only +bin/doclinks docs/guide.md # single file +bin/doclinks docs/ # directory +bin/doclinks --dry-run ... # preview only ``` ## Full Documentation @@ -60,26 +60,26 @@ See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. ```bash # Process a single file -doclinks docs/guide.md +bin/doclinks docs/guide.md # Process a directory recursively -doclinks docs/ +bin/doclinks docs/ # Relative links (from doc location) -doclinks --link-mode relative docs/ +bin/doclinks --link-mode relative docs/ # GitHub links -doclinks --link-mode github \ +bin/doclinks --link-mode github \ --github-url https://github.com/org/repo docs/ # Dry run (preview changes) -doclinks --dry-run docs/ +bin/doclinks --dry-run docs/ # CI check (exit 1 if changes needed) -doclinks --check docs/ +bin/doclinks --check docs/ # Watch mode (auto-update on changes) -doclinks --watch docs/ +bin/doclinks --watch docs/ ``` ## Options From 83aa40841b14b571365404cb88691c35705c95fa Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 11:45:31 -0800 Subject: [PATCH 104/136] remove stuff stash didn't like --- README.md | 202 ------------------------------------------------------ 1 file changed, 202 deletions(-) diff --git a/README.md b/README.md index aea2aa0803..76fd32e57d 100644 --- a/README.md +++ b/README.md @@ -109,208 +109,6 @@ humancli We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. -# How do I use it as a library? - -### Simple Camera Activation - -Assuming you have a webcam, save the following as a python file and run it: - -```py -from dimos.core.blueprints import autoconnect -from dimos.hardware.sensors.camera.module import CameraModule - -if __name__ == "__main__": - autoconnect( - # technically autoconnect is not needed because we only have 1 module - CameraModule.blueprint() - ).build().loop() -``` - -### Write A Custom Module - -Lets convert the camera's image to grayscale. - -```py -from dimos.core.blueprints import autoconnect -from dimos.core import In, Out, Module, rpc -from dimos.hardware.sensors.camera.module import CameraModule -from dimos.msgs.sensor_msgs import Image - -from reactivex.disposable import Disposable - -class Listener(Module): - # the CameraModule has an Out[Image] named "color_image" - # How do we know this? Just print(CameraModule.module_info().outputs) - # the name ("color_image") must match the CameraModule's output - color_image: In[Image] = None - grayscale_image: Out[Image] = None - - def __init__(self, *args, **kwargs) -> None: - super().__init__(*args, **kwargs) - self.count = 0 - - @rpc - def start(self) -> None: - super().start() - def callback_func(img: Image) -> None: - self.count += 1 - print(f"got frame {self.count}") - print(f"img.data.shape: {img.data.shape}") - self.grayscale_image.publish(img.to_grayscale()) - - unsubscribe_func = self.color_image.subscribe(callback_func) - # the unsubscribe_func be called when the module is stopped - self._disposables.add(Disposable( - unsubscribe_func - )) - - @rpc - def stop(self) -> None: - super().stop() - -if __name__ == "__main__": - autoconnect( - Listener.blueprint(), - CameraModule.blueprint(), - ).build().loop() -``` - -#### RPC calls between modules - -Call another module's `@rpc` method by listing it in `rpc_calls` and grabbing it with `get_rpc_calls`. - -```py -from dimos.core import Module, rpc -from dimos.core.blueprints import autoconnect - - -class Counter(Module): - def __init__(self) -> None: - super().__init__() - self._count = 0 - - @rpc - def start(self) -> None: - super().start() - print("Counter ready") - - @rpc - def add(self, amount: int = 1) -> int: - """Add to the running total and return the new count.""" - self._count += amount - print(f"Counter now {self._count}") - return self._count - - @rpc - def stop(self) -> None: - super().stop() - - -class Client(Module): - # ask for the Counter.add RPC, then call it from start() - rpc_calls = ["Counter.add"] - - @rpc - def start(self) -> None: - super().start() - counter_add = self.get_rpc_calls("Counter.add") - counter_add(2) - final_value = counter_add(3) - print(f"Remote counter ended at {final_value}") - - @rpc - def stop(self) -> None: - super().stop() - - -if __name__ == "__main__": - autoconnect( - Counter.blueprint(), - Client.blueprint(), - ).build().loop() -``` - -#### Skills in a module - -Define skills with `@skill`, register them with a `SkillCoordinator`, and retrieve the results as they stream back. - -```py -import asyncio - -from dimos.core import Module, rpc -from dimos.protocol.skill.coordinator import SkillCoordinator -from dimos.protocol.skill.skill import skill -from dimos.protocol.skill.type import Reducer, Stream - - -class Calculator(Module): - @rpc - def start(self) -> None: - super().start() - - @rpc - def stop(self) -> None: - super().stop() - - @skill() - def add(self, x: int, y: int) -> int: - return x + y - - @skill(stream=Stream.passive, reducer=Reducer.latest) # type: ignore[arg-type] - def count_to(self, n: int): - for i in range(1, n + 1): - yield i - - -async def main() -> None: - skills = Calculator() - coordinator = SkillCoordinator() - coordinator.register_skills(skills) - coordinator.start() - - coordinator.call_skill("add-1", "add", {"args": [2, 3]}) - coordinator.call_skill("count-1", "count_to", {"args": [4]}) - - while await coordinator.wait_for_updates(timeout=1): - updates = coordinator.generate_snapshot(clear=True) - for call_id, state in updates.items(): - print(f"{call_id} -> {state.content()}") - if not coordinator.has_passive_skills(): - break - - coordinator.stop() - - -if __name__ == "__main__": - asyncio.run(main()) -``` - -#### Note: Many More Examples in the [Examples Folder](./examples) - -### How do custom modules work? (Example breakdown) - -- Every module represents one process: modules run in parallel (python multiprocessing). Because of this **modules should only save/modify data on themselves**. Do not mutate or share global vars inside a module. -- At the top of this module definition, the In/Out **streams** are defining a pub-sub system. This module expects *someone somewhere* to give it a color image. And, the module is going to publish a grayscale image (that any other module to subscribe to). - - Note: if you are a power user thinking "so streams must be statically declared?" the answer is no, there are ways to perform dynamic connections, but for type-checking and human sanity the creation of dynamic stream connections are under an advanced API and should be used as a last resort. -- The `autoconnect` ties everything together: - - The CameraModule has an output of `color_image` - - The Listener has an input of `color_image` - - Autoconnect puts them together, and checks that their types are compatible (both are of type `Image`) -- How can we see what In/Out streams are provided by a module? - - Open a python repl (e.g. `python`) - - Import the module, ex: `from dimos.hardware.sensors.camera.module import CameraModule` - - Print the module outputs: `print(CameraModule.module_info().outputs)` - - Print the module inputs: `print(CameraModule.module_info().inputs)` - - Print all the information (rpcs, skills, etc): `print(CameraModule.module_info())` -- What about `@rpc`? - - If you want a method to be called by another module (not just an internal method) then add the `@rpc` decorator AND make sure BOTH the arguments and return value of the method are json-serializable. - - The start/stop methods always need to be an rpc because they are called externally. - - Under the hood rpc methods get called using threads. This means it is possible for two rpc methods to be running at the same time. For this reason, python thread locking is often necessary for data that is being written/read during rpc calls. - - To call another module's RPC, add its name to `rpc_calls` (ex: `["Counter.add"]`), then `self.get_rpc_calls("Counter.add")` returns a callable that forwards across transports. The method key is always `ClassName.method_name`. Note: this API will likely be replaced with a more convenient API. -- What about skills? - - Skills are upgraded rpc methods designed to be called by an AI agent. - - Because skills can be dynamically added/removed, they are registered with a SkillCoordinator. - ### Documentation & Concepts If you you need more information on how DimOS works, check out the following links: From c920da85f0400df561d97ea334cd409847017b55 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 12:00:28 -0800 Subject: [PATCH 105/136] hide TODO in a comment --- docs/concepts/modules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index ee7fbaf2c9..dc42940e84 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -120,7 +120,7 @@ print(Detection2DModule.io()) ├─ RPC stop() -> None ``` -TODO: add easy way to print config + Looks like the detector just needs an image input and outputs some sort of detection and annotation messages. Let's connect it to a camera. From 2c358b7c66646bcf9f1dbf8bb42f54ced770f12d Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 12:11:05 -0800 Subject: [PATCH 106/136] add a link to blueprints --- docs/concepts/modules.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index dc42940e84..78cf288ae1 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -174,3 +174,6 @@ to_svg(agentic, "assets/go2_agentic.svg") ![output](assets/go2_agentic.svg) + + +To see more information on how to use Blueprints, see [Blueprints](/docs/concepts/blueprints.md). From 54d65e2401b0861042efa5df9a5914586af0ada7 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 13:07:09 -0800 Subject: [PATCH 107/136] minor change --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7cbc884608..dcf5a4e8a7 100644 --- a/README.md +++ b/README.md @@ -139,4 +139,4 @@ In addition to rerun logging, DimOS comes with a number of monitoring tools: # License -DimOS is licensed under the Apache License, Version 2.0. And will always be free and open source. +DimOS is licensed under the Apache License, Version 2.0. From 4ecb0243e9cbdbe9646fa19071d7e22a2be1061c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:01:41 -0800 Subject: [PATCH 108/136] use different examples in readme --- README.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 113 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index dcf5a4e8a7..778341f823 100644 --- a/README.md +++ b/README.md @@ -114,9 +114,7 @@ source .venv/bin/activate humancli ``` -## Contributing / Building From Source - -We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. +## Using DimOS as a Library ### Documentation & Concepts @@ -129,6 +127,118 @@ If you you need more information on how DimOS works, check out the following lin - [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. +### Simple DimOS Application + +While `dimos run unitree-go2` is a good demo, we can recreate it with a few lines of code. + +Save and run the following to get behavior like `dimos run unitree-go2`: + +```python +from dimos.robot.unitree_webrtc.unitree_go2_blueprints import autoconnect, basic, cost_mapper, voxel_mapper, replanning_a_star_planner, wavefront_frontier_explorer + +nav = autoconnect( + basic, + voxel_mapper(voxel_size=0.3), + cost_mapper(), +).global_config(simulation=True) + +nav.build().loop() +``` + + +#### Printing Module I/O + +If you're not sure how to use a module, start by printing its I/O: + +```python session=camera_module_demo ansi=false +from dimos.hardware.sensors.camera.module import CameraModule +from dimos.perception.detection.module2D import Detection2DModule + +print("Camera:\n", CameraModule.io()) +print("Detection:\n", Detection2DModule.io()) +``` + + +``` +Camera: +┌┴─────────────┐ +│ CameraModule │ +└┬─────────────┘ + ├─ color_image: Image + ├─ camera_info: CameraInfo + │ + ├─ RPC start() + ├─ RPC stop() + │ + ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) + + Detection: + ├─ image: Image +┌┴──────────────────┐ +│ Detection2DModule │ +└┬──────────────────┘ + ├─ detections: Detection2DArray + ├─ annotations: ImageAnnotations + ├─ detected_image_0: Image + ├─ detected_image_1: Image + ├─ detected_image_2: Image + │ + ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool + ├─ RPC start() -> None + ├─ RPC stop() -> None +``` + +We will talk about Skills and RPC calls later, the main thing above is that: +1. The CameraModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) output stream. +2. The Detection2DModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) input stream. + + +#### Connecting Modules Manually + +Using a module basically always means connecting it in some way, so let's connect those two modules. + +```python session=camera_module_demo ansi=false +import time +camera = CameraModule() +detector = Detection2DModule() + +detector.image.connect(camera.color_image) + +camera.start() +detector.start() + +detector.detections.subscribe(print) +time.sleep(3) +detector.stop() +camera.stop() +``` + + +``` +Detection(Person(1)) +Detection(Person(1)) +Detection(Person(1)) +Detection(Person(1)) +``` + +#### Connecting Modules Automatically + +While modules can be connected individually, it would be nice to pre-connect some parts while allowing others to override or extend those connections. For that we have **Blueprints**. + +```python ansi=false +autoconnect( + CameraModule.blueprint(), + Detection2DModule.blueprint(), +).build().loop() +``` + + + +## Contributing / Building From Source + +We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. + + ### Monitoring & Debugging In addition to rerun logging, DimOS comes with a number of monitoring tools: From ddd0575fbae589a4f7d2954b1ad6f0c1bf1360c2 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:02:15 -0800 Subject: [PATCH 109/136] revert using different examples --- README.md | 116 ++---------------------------------------------------- 1 file changed, 3 insertions(+), 113 deletions(-) diff --git a/README.md b/README.md index 778341f823..dcf5a4e8a7 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,9 @@ source .venv/bin/activate humancli ``` -## Using DimOS as a Library +## Contributing / Building From Source + +We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. ### Documentation & Concepts @@ -127,118 +129,6 @@ If you you need more information on how DimOS works, check out the following lin - [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. -### Simple DimOS Application - -While `dimos run unitree-go2` is a good demo, we can recreate it with a few lines of code. - -Save and run the following to get behavior like `dimos run unitree-go2`: - -```python -from dimos.robot.unitree_webrtc.unitree_go2_blueprints import autoconnect, basic, cost_mapper, voxel_mapper, replanning_a_star_planner, wavefront_frontier_explorer - -nav = autoconnect( - basic, - voxel_mapper(voxel_size=0.3), - cost_mapper(), -).global_config(simulation=True) - -nav.build().loop() -``` - - -#### Printing Module I/O - -If you're not sure how to use a module, start by printing its I/O: - -```python session=camera_module_demo ansi=false -from dimos.hardware.sensors.camera.module import CameraModule -from dimos.perception.detection.module2D import Detection2DModule - -print("Camera:\n", CameraModule.io()) -print("Detection:\n", Detection2DModule.io()) -``` - - -``` -Camera: -┌┴─────────────┐ -│ CameraModule │ -└┬─────────────┘ - ├─ color_image: Image - ├─ camera_info: CameraInfo - │ - ├─ RPC start() - ├─ RPC stop() - │ - ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) - - Detection: - ├─ image: Image -┌┴──────────────────┐ -│ Detection2DModule │ -└┬──────────────────┘ - ├─ detections: Detection2DArray - ├─ annotations: ImageAnnotations - ├─ detected_image_0: Image - ├─ detected_image_1: Image - ├─ detected_image_2: Image - │ - ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool - ├─ RPC start() -> None - ├─ RPC stop() -> None -``` - -We will talk about Skills and RPC calls later, the main thing above is that: -1. The CameraModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) output stream. -2. The Detection2DModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) input stream. - - -#### Connecting Modules Manually - -Using a module basically always means connecting it in some way, so let's connect those two modules. - -```python session=camera_module_demo ansi=false -import time -camera = CameraModule() -detector = Detection2DModule() - -detector.image.connect(camera.color_image) - -camera.start() -detector.start() - -detector.detections.subscribe(print) -time.sleep(3) -detector.stop() -camera.stop() -``` - - -``` -Detection(Person(1)) -Detection(Person(1)) -Detection(Person(1)) -Detection(Person(1)) -``` - -#### Connecting Modules Automatically - -While modules can be connected individually, it would be nice to pre-connect some parts while allowing others to override or extend those connections. For that we have **Blueprints**. - -```python ansi=false -autoconnect( - CameraModule.blueprint(), - Detection2DModule.blueprint(), -).build().loop() -``` - - - -## Contributing / Building From Source - -We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. - - ### Monitoring & Debugging In addition to rerun logging, DimOS comes with a number of monitoring tools: From 4ba47481bed131d2f9346aa0092c16f47cfc401c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:11:29 -0800 Subject: [PATCH 110/136] update diagram docs --- .../writing_docs/diagram_practices.md | 169 +++++++++--------- 1 file changed, 85 insertions(+), 84 deletions(-) diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index 7a27d89b9b..f538fede8a 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -1,131 +1,132 @@ We have many diagraming tools. View source code of this page to see examples. -### Pikchr +# Pikchr -SQLite's diagram language: +[Pikchr](https://pikchr.org/) is a diagram language from SQLite. Use it for flowcharts and architecture diagrams. + +**Important:** Always wrap pikchr blocks in `
` tags so the source is collapsed by default on GitHub. The rendered SVG stays visible outside the fold. Code blocks (Python, etc.) should NOT be folded—they're meant to be read. + +## Basic syntax
diagram source -```pikchr fold output=assets/pikchr-demo.svg +```pikchr fold output=assets/pikchr_basic.svg color = white fill = none -linewid = 0.4in - -# Input file -In: file "README.md" fit -arrow -# Processing -Parse: box "Parse" rad 5px fit -arrow -Exec: box "Execute" rad 5px fit - -# Fan out to languages -arrow from Exec.e right 0.3in then up 0.4in then right 0.3in -Sh: oval "Shell" fit -arrow from Exec.e right 0.3in then right 0.3in -Node: oval "Node" fit -arrow from Exec.e right 0.3in then down 0.4in then right 0.3in -Py: oval "Python" fit - -# Merge back -X: dot at (Py.e.x + 0.3in, Node.e.y) invisible -line from Sh.e right until even with X then down to X -line from Node.e to X -line from Py.e right until even with X then up to X -Out: file "README.md" fit with .w at (X.x + 0.3in, X.y) -arrow from X to Out.w +A: box "Step 1" rad 5px fit wid 170% ht 170% +arrow right 0.3in +B: box "Step 2" rad 5px fit wid 170% ht 170% +arrow right 0.3in +C: box "Step 3" rad 5px fit wid 170% ht 170% ```
-![output](assets/pikchr-demo.svg) +![output](assets/pikchr_basic.svg) -### Asymptote +## Box sizing -Vector graphics: +Use `fit` with percentage scaling to auto-size boxes with padding: -```asymptote output=assets/histogram.svg -import graph; -import stats; +
+diagram source -size(400,200,IgnoreAspect); -defaultpen(white); +```pikchr fold output=assets/pikchr_sizing.svg +color = white +fill = none -int n=10000; -real[] a=new real[n]; -for(int i=0; i < n; ++i) a[i]=Gaussrand(); +# fit wid 170% ht 170% = auto-size + padding +A: box "short" rad 5px fit wid 170% ht 170% +arrow right 0.3in +B: box ".subscribe()" rad 5px fit wid 170% ht 170% +arrow right 0.3in +C: box "two lines" "of text" rad 5px fit wid 170% ht 170% +``` -draw(graph(Gaussian,min(a),max(a)),orange); +
-int N=bins(a); + +![output](assets/pikchr_sizing.svg) -histogram(a,min(a),max(a),N,normalize=true,low=0,rgb(0.4,0.6,0.8),rgb(0.2,0.4,0.6),bars=true); +The pattern `fit wid 170% ht 170%` means: auto-size to text, then scale width by 170% and height by 170%. -xaxis("$x$",BottomTop,LeftTicks,p=white); -yaxis("$dP/dx$",LeftRight,RightTicks(trailingzero),p=white); -``` +For explicit sizing (when you need consistent box sizes): - -![output](assets/histogram.svg) +
+diagram source -### Graphviz +```pikchr fold output=assets/pikchr_explicit.svg +color = white +fill = none -```dot output=assets/graph.svg -A -> B -> C -A -> C +A: box "Step 1" rad 5px fit wid 170% ht 170% +arrow right 0.3in +B: box "Step 2" rad 5px fit wid 170% ht 170% ``` +
+ -![output](assets/graph.svg) +![output](assets/pikchr_explicit.svg) + +## Common settings -### OpenSCAD +Always start with: -```openscad output=assets/cube-sphere.png -cube([10, 10, 10]); -sphere(r=7); +``` +color = white # text color +fill = none # transparent box fill ``` - -![output](assets/cube-sphere.png) +## Branching paths -### Diagon +
+diagram source + +```pikchr fold output=assets/pikchr_branch.svg +color = white +fill = none + +A: box "Input" rad 5px fit wid 170% ht 170% +arrow +B: box "Process" rad 5px fit wid 170% ht 170% -ASCII art diagrams: +# Branch up +arrow from B.e right 0.3in then up 0.35in then right 0.3in +C: box "Path A" rad 5px fit wid 170% ht 170% -```diagon mode=Math -1 + 1/2 + sum(i,0,10) +# Branch down +arrow from B.e right 0.3in then down 0.35in then right 0.3in +D: box "Path B" rad 5px fit wid 170% ht 170% ``` +
+ -``` - 10 - ___ - 1 ╲ -1 + ─ + ╱ i - 2 ‾‾‾ - 0 -``` +![output](assets/pikchr_branch.svg) + +**Tip:** For tree/hierarchy diagrams, prefer left-to-right layout (root on left, children branching right). This reads more naturally and avoids awkward vertical stacking. -```diagon mode=GraphDAG -A -> B -> C -A -> C +## Adding labels + +
+diagram source + +```pikchr fold output=assets/pikchr_labels.svg +color = white +fill = none + +A: box "Box" rad 5px fit wid 170% ht 170% +text "label below" at (A.x, A.y - 0.4in) ``` +
+ -``` -┌───┐ -│A │ -└┬─┬┘ - │┌▽┐ - ││B│ - │└┬┘ -┌▽─▽┐ -│C │ -└───┘ -``` +![output](assets/pikchr_labels.svg) ## Reference From 6c67199feafa5f375eba566835f7caea1979cf89 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:26:21 -0800 Subject: [PATCH 111/136] rename back --- docs/development/writing_docs/{code_blocks.md => codeblocks.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/development/writing_docs/{code_blocks.md => codeblocks.md} (100%) diff --git a/docs/development/writing_docs/code_blocks.md b/docs/development/writing_docs/codeblocks.md similarity index 100% rename from docs/development/writing_docs/code_blocks.md rename to docs/development/writing_docs/codeblocks.md From ee3635f795340f8648bd2558674fa02212d38633 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:28:29 -0800 Subject: [PATCH 112/136] fix diagrams --- docs/development/assets/pikchr_basic.svg | 12 ------------ docs/development/assets/pikchr_sizing.svg | 13 ------------- .../assets/codeblocks_example.svg | 0 .../writing_docs/assets/pikchr_basic.svg | 12 ++++++++++++ .../{ => writing_docs}/assets/pikchr_branch.svg | 0 .../{ => writing_docs}/assets/pikchr_explicit.svg | 0 .../{ => writing_docs}/assets/pikchr_labels.svg | 0 .../writing_docs/assets/pikchr_sizing.svg | 13 +++++++++++++ 8 files changed, 25 insertions(+), 25 deletions(-) delete mode 100644 docs/development/assets/pikchr_basic.svg delete mode 100644 docs/development/assets/pikchr_sizing.svg rename docs/development/{ => writing_docs}/assets/codeblocks_example.svg (100%) create mode 100644 docs/development/writing_docs/assets/pikchr_basic.svg rename docs/development/{ => writing_docs}/assets/pikchr_branch.svg (100%) rename docs/development/{ => writing_docs}/assets/pikchr_explicit.svg (100%) rename docs/development/{ => writing_docs}/assets/pikchr_labels.svg (100%) create mode 100644 docs/development/writing_docs/assets/pikchr_sizing.svg diff --git a/docs/development/assets/pikchr_basic.svg b/docs/development/assets/pikchr_basic.svg deleted file mode 100644 index 5410d35577..0000000000 --- a/docs/development/assets/pikchr_basic.svg +++ /dev/null @@ -1,12 +0,0 @@ - - -Step 1 - - - -Step 2 - - - -Step 3 - diff --git a/docs/development/assets/pikchr_sizing.svg b/docs/development/assets/pikchr_sizing.svg deleted file mode 100644 index 3a0c433cb1..0000000000 --- a/docs/development/assets/pikchr_sizing.svg +++ /dev/null @@ -1,13 +0,0 @@ - - -short - - - -.subscribe() - - - -two lines -of text - diff --git a/docs/development/assets/codeblocks_example.svg b/docs/development/writing_docs/assets/codeblocks_example.svg similarity index 100% rename from docs/development/assets/codeblocks_example.svg rename to docs/development/writing_docs/assets/codeblocks_example.svg diff --git a/docs/development/writing_docs/assets/pikchr_basic.svg b/docs/development/writing_docs/assets/pikchr_basic.svg new file mode 100644 index 0000000000..9dee0bfdec --- /dev/null +++ b/docs/development/writing_docs/assets/pikchr_basic.svg @@ -0,0 +1,12 @@ + + +Step 1 + + + +Step 2 + + + +Step 3 + diff --git a/docs/development/assets/pikchr_branch.svg b/docs/development/writing_docs/assets/pikchr_branch.svg similarity index 100% rename from docs/development/assets/pikchr_branch.svg rename to docs/development/writing_docs/assets/pikchr_branch.svg diff --git a/docs/development/assets/pikchr_explicit.svg b/docs/development/writing_docs/assets/pikchr_explicit.svg similarity index 100% rename from docs/development/assets/pikchr_explicit.svg rename to docs/development/writing_docs/assets/pikchr_explicit.svg diff --git a/docs/development/assets/pikchr_labels.svg b/docs/development/writing_docs/assets/pikchr_labels.svg similarity index 100% rename from docs/development/assets/pikchr_labels.svg rename to docs/development/writing_docs/assets/pikchr_labels.svg diff --git a/docs/development/writing_docs/assets/pikchr_sizing.svg b/docs/development/writing_docs/assets/pikchr_sizing.svg new file mode 100644 index 0000000000..9d17f571d1 --- /dev/null +++ b/docs/development/writing_docs/assets/pikchr_sizing.svg @@ -0,0 +1,13 @@ + + +short + + + +.subscribe() + + + +two lines +of text + From 1cc7d1a7425b3a5919aa1fd02bfc7699776100b6 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:31:57 -0800 Subject: [PATCH 113/136] add instruction --- docs/development/writing_docs/diagram_practices.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index f538fede8a..44bed0447c 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -1,5 +1,10 @@ We have many diagraming tools. View source code of this page to see examples. +# How to make diagrams + +1. Use Pikchr (language) to define a diagram +2. Use `md-babel-py` from [codeblocks.md](/docs/development/writing_docs/codeblocks.md) to execute the diagram and insert the result + # Pikchr [Pikchr](https://pikchr.org/) is a diagram language from SQLite. Use it for flowcharts and architecture diagrams. From 476aae3d6ba79a1a979e63d45b5b0aab3a7c6551 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:44:12 -0800 Subject: [PATCH 114/136] revert revert --- README.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 113 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index dcf5a4e8a7..778341f823 100644 --- a/README.md +++ b/README.md @@ -114,9 +114,7 @@ source .venv/bin/activate humancli ``` -## Contributing / Building From Source - -We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. +## Using DimOS as a Library ### Documentation & Concepts @@ -129,6 +127,118 @@ If you you need more information on how DimOS works, check out the following lin - [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). - Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. +### Simple DimOS Application + +While `dimos run unitree-go2` is a good demo, we can recreate it with a few lines of code. + +Save and run the following to get behavior like `dimos run unitree-go2`: + +```python +from dimos.robot.unitree_webrtc.unitree_go2_blueprints import autoconnect, basic, cost_mapper, voxel_mapper, replanning_a_star_planner, wavefront_frontier_explorer + +nav = autoconnect( + basic, + voxel_mapper(voxel_size=0.3), + cost_mapper(), +).global_config(simulation=True) + +nav.build().loop() +``` + + +#### Printing Module I/O + +If you're not sure how to use a module, start by printing its I/O: + +```python session=camera_module_demo ansi=false +from dimos.hardware.sensors.camera.module import CameraModule +from dimos.perception.detection.module2D import Detection2DModule + +print("Camera:\n", CameraModule.io()) +print("Detection:\n", Detection2DModule.io()) +``` + + +``` +Camera: +┌┴─────────────┐ +│ CameraModule │ +└┬─────────────┘ + ├─ color_image: Image + ├─ camera_info: CameraInfo + │ + ├─ RPC start() + ├─ RPC stop() + │ + ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) + + Detection: + ├─ image: Image +┌┴──────────────────┐ +│ Detection2DModule │ +└┬──────────────────┘ + ├─ detections: Detection2DArray + ├─ annotations: ImageAnnotations + ├─ detected_image_0: Image + ├─ detected_image_1: Image + ├─ detected_image_2: Image + │ + ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool + ├─ RPC start() -> None + ├─ RPC stop() -> None +``` + +We will talk about Skills and RPC calls later, the main thing above is that: +1. The CameraModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) output stream. +2. The Detection2DModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) input stream. + + +#### Connecting Modules Manually + +Using a module basically always means connecting it in some way, so let's connect those two modules. + +```python session=camera_module_demo ansi=false +import time +camera = CameraModule() +detector = Detection2DModule() + +detector.image.connect(camera.color_image) + +camera.start() +detector.start() + +detector.detections.subscribe(print) +time.sleep(3) +detector.stop() +camera.stop() +``` + + +``` +Detection(Person(1)) +Detection(Person(1)) +Detection(Person(1)) +Detection(Person(1)) +``` + +#### Connecting Modules Automatically + +While modules can be connected individually, it would be nice to pre-connect some parts while allowing others to override or extend those connections. For that we have **Blueprints**. + +```python ansi=false +autoconnect( + CameraModule.blueprint(), + Detection2DModule.blueprint(), +).build().loop() +``` + + + +## Contributing / Building From Source + +We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. + + ### Monitoring & Debugging In addition to rerun logging, DimOS comes with a number of monitoring tools: From bbc1fee9924e85598c68e43829513d6359d1ad06 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 15:52:32 -0800 Subject: [PATCH 115/136] - --- docs/development/writing_docs/diagram_practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index 44bed0447c..ef4eb0ce8a 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -3,7 +3,7 @@ We have many diagraming tools. View source code of this page to see examples. # How to make diagrams 1. Use Pikchr (language) to define a diagram -2. Use `md-babel-py` from [codeblocks.md](/docs/development/writing_docs/codeblocks.md) to execute the diagram and insert the result +2. Use `md-babel-py` from [codeblocks.md](/docs/development/writing_docs/codeblocks.md) to execute the diagram and insert the result. # Pikchr From 8d04d50746cca6751824129fe7f80859d0228f0a Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 23:28:46 -0600 Subject: [PATCH 116/136] fix link Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 778341f823..2c5e722425 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ humancli If you you need more information on how DimOS works, check out the following links: - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. -- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Streams](/docs/api/sensor_streams/README.md): How modules communicate, a Pub / Sub system. - [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. - [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). - [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). From 145a2a02c4ba31679d3e2d91e1e1a843b3387ca1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 23:29:18 -0600 Subject: [PATCH 117/136] improve link Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- docs/concepts/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/README.md b/docs/concepts/README.md index 93b25cde5f..7118467f70 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -1,6 +1,6 @@ # Concepts -If you are trying to learn a new API or know what tools you should use, then you are in the right place. In contrast, if you already know the API you want to use and just need details on how to use it, see [the API reference](/docs/API/README.md). +If you are trying to learn a new API or know what tools you should use, then you are in the right place. In contrast, if you already know the API you want to use and just need details on how to use it, see [the API reference](/docs/api/README.md). ## Table of Contents From 94b5e9415765475ada9c7ee42de39ca573d61cc3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 23:29:29 -0600 Subject: [PATCH 118/136] fix link Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- docs/concepts/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/README.md b/docs/concepts/README.md index 7118467f70..ed5c00ab01 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -5,7 +5,7 @@ If you are trying to learn a new API or know what tools you should use, then you ## Table of Contents - [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. -- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system. +- [Streams](/docs/api/sensor_streams/README.md): How modules communicate, a Pub / Sub system. - [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. - [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). - [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). From fc622aeacf9b7a4f23227721ea651e30e8ff13c3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 23:29:47 -0600 Subject: [PATCH 119/136] fix link Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index b914403387..9a6f296fbc 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -266,7 +266,7 @@ You can enable a tag by selecting -m - these are configured in `./pyp - Open the PR against the `dev` branch (not `main`). - **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. -- If you're writing documentation, see [writing docs](/docs/development/writing_docs.md) +- If you're writing documentation, see [writing docs](/docs/development/writing_docs/README.md) - If you get mypy errors, please fix them. Don't just add # type: ignore. Please first understand why mypy is complaining and try to fix it. It's only okay to ignore if the issue cannot be fixed. - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). From 95c04d8459a80ba768f404d1f3304fd25a86032e Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 21:38:00 -0800 Subject: [PATCH 120/136] remove all but the "Simple DimOS Application" --- README.md | 89 ------------------------------------------------------- 1 file changed, 89 deletions(-) diff --git a/README.md b/README.md index 2c5e722425..74b8e28b6c 100644 --- a/README.md +++ b/README.md @@ -145,95 +145,6 @@ nav = autoconnect( nav.build().loop() ``` - -#### Printing Module I/O - -If you're not sure how to use a module, start by printing its I/O: - -```python session=camera_module_demo ansi=false -from dimos.hardware.sensors.camera.module import CameraModule -from dimos.perception.detection.module2D import Detection2DModule - -print("Camera:\n", CameraModule.io()) -print("Detection:\n", Detection2DModule.io()) -``` - - -``` -Camera: -┌┴─────────────┐ -│ CameraModule │ -└┬─────────────┘ - ├─ color_image: Image - ├─ camera_info: CameraInfo - │ - ├─ RPC start() - ├─ RPC stop() - │ - ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) - - Detection: - ├─ image: Image -┌┴──────────────────┐ -│ Detection2DModule │ -└┬──────────────────┘ - ├─ detections: Detection2DArray - ├─ annotations: ImageAnnotations - ├─ detected_image_0: Image - ├─ detected_image_1: Image - ├─ detected_image_2: Image - │ - ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool - ├─ RPC start() -> None - ├─ RPC stop() -> None -``` - -We will talk about Skills and RPC calls later, the main thing above is that: -1. The CameraModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) output stream. -2. The Detection2DModule provides an [Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) input stream. - - -#### Connecting Modules Manually - -Using a module basically always means connecting it in some way, so let's connect those two modules. - -```python session=camera_module_demo ansi=false -import time -camera = CameraModule() -detector = Detection2DModule() - -detector.image.connect(camera.color_image) - -camera.start() -detector.start() - -detector.detections.subscribe(print) -time.sleep(3) -detector.stop() -camera.stop() -``` - - -``` -Detection(Person(1)) -Detection(Person(1)) -Detection(Person(1)) -Detection(Person(1)) -``` - -#### Connecting Modules Automatically - -While modules can be connected individually, it would be nice to pre-connect some parts while allowing others to override or extend those connections. For that we have **Blueprints**. - -```python ansi=false -autoconnect( - CameraModule.blueprint(), - Detection2DModule.blueprint(), -).build().loop() -``` - - - ## Contributing / Building From Source We welcome contributions! Open up the [Development Guide](/docs/development/README.md) to see how to hack on DimOS and make PR's and our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue. From 62080b6076c53e6f69bca9920220f6874e7e6850 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 21:47:04 -0800 Subject: [PATCH 121/136] lots of minor fixes --- docs/api/configuration.md | 2 +- docs/api/sensor_streams/quality_filter.md | 2 +- docs/api/sensor_streams/storage_replay.md | 2 +- docs/api/sensor_streams/temporal_alignment.md | 4 ++-- docs/api/visualization.md | 2 +- docs/concepts/modules.md | 6 +++--- docs/concepts/transports.md | 14 ++++++++++---- docs/development/README.md | 12 ++++++------ docs/development/depth_camera_integration.md | 12 ++++++------ docs/development/writing_docs/README.md | 6 +++--- docs/development/writing_docs/diagram_practices.md | 2 +- 11 files changed, 35 insertions(+), 29 deletions(-) diff --git a/docs/api/configuration.md b/docs/api/configuration.md index a9c8de0268..162c343ffa 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids, etc. +[Modules](/docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame IDs, etc. ```python from dataclasses import dataclass diff --git a/docs/api/sensor_streams/quality_filter.md b/docs/api/sensor_streams/quality_filter.md index 26d40733fd..01d3b9367d 100644 --- a/docs/api/sensor_streams/quality_filter.md +++ b/docs/api/sensor_streams/quality_filter.md @@ -50,7 +50,7 @@ Qualities: [0.9] For camera streams, we provide `sharpness_barrier` which uses the image's sharpness score. -Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Replay](/docs/old/testing_stream_reply.md) toolkit, which provides access to recorded robot data: +Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Storage & Replay](/docs/api/sensor_streams/storage_replay.md) toolkit, which provides access to recorded robot data: ```python session=qb from dimos.utils.testing import TimedSensorReplay diff --git a/docs/api/sensor_streams/storage_replay.md b/docs/api/sensor_streams/storage_replay.md index 66e913b197..1a31591736 100644 --- a/docs/api/sensor_streams/storage_replay.md +++ b/docs/api/sensor_streams/storage_replay.md @@ -202,7 +202,7 @@ Each pickle file contains a tuple `(timestamp, data)`: Files are numbered sequentially: `000.pickle`, `001.pickle`, etc. -Recordings are stored in the `data/` directory. See [Data Loading](/docs/data.md) for how data storage works, including Git LFS handling for large datasets. +Recordings are stored in the `data/` directory. See [Data Loading](/docs/development/large_file_management.md) for how data storage works, including Git LFS handling for large datasets. ## API Reference diff --git a/docs/api/sensor_streams/temporal_alignment.md b/docs/api/sensor_streams/temporal_alignment.md index b552ac54cc..66230c9d54 100644 --- a/docs/api/sensor_streams/temporal_alignment.md +++ b/docs/api/sensor_streams/temporal_alignment.md @@ -34,7 +34,7 @@ Below we set up replay of real camera and lidar data from the Unitree Go2 robot.
Stream Setup -You can read more about [sensor storage here](storage_replay.md) and [LFS data store here](/docs/data.md). +You can read more about [sensor storage here](storage_replay.md) and [LFS data storage here](/docs/development/large_file_management.md). ```python session=align no-result from reactivex import Subject @@ -70,7 +70,7 @@ lidar_stream = lidar_replay.stream(from_timestamp=seek_ts, duration=2.0).pipe(
-Streams would normally come from an actual robot into your module via `IN` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. +Streams would normally come from an actual robot into your module via `In` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. Assume we have them. Let's align them. diff --git a/docs/api/visualization.md b/docs/api/visualization.md index 51fa20d655..08919b7a73 100644 --- a/docs/api/visualization.md +++ b/docs/api/visualization.md @@ -54,7 +54,7 @@ VIEWER_BACKEND=foxglove dimos run unitree-go2 - Foxglove bridge on ws://localhost:8765 - No Rerun (saves resources) - Better performance with larger maps/higher resolution -- Open layout: `dimos/assets/foxglove_dashboards/go2.json` +- Open layout: `assets/foxglove_dashboards/old/foxglove_unitree_lcm_dashboard.json` --- diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index 78cf288ae1..6f167cb43e 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -1,5 +1,5 @@ -# Dimos Modules +# DimOS Modules Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages. @@ -58,9 +58,9 @@ We can see that the camera module outputs two streams: - `color_image` with [sensor_msgs.Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) type - `camera_info` with [sensor_msgs.CameraInfo](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) type -It offers two RPC calls: `start()` and `stop()`. +It offers one module-specific RPC call: `start()`. You will also see the base `set_transport(...)` RPC from `Module`. The module still has a `stop()` lifecycle method, but it's called on the local instance (or by the coordinator), not via RPC. -As well as an agentic [Skill](skills.md) called `video_stream` (more about this later, in [Skills Tutorial](skills.md)). +It also exposes an agentic [skill](/docs/concepts/blueprints.md#defining-skills) called `video_stream` (more on skills in the Blueprints guide). We can start this module and explore the output of its streams in real time (this will use your webcam). diff --git a/docs/concepts/transports.md b/docs/concepts/transports.md index 17780a2b2e..6f37f8c631 100644 --- a/docs/concepts/transports.md +++ b/docs/concepts/transports.md @@ -18,11 +18,17 @@ print(inspect.getsource(PubSub.publish)) print(inspect.getsource(PubSub.subscribe)) ``` - ``` -Session process exited unexpectedly: -/home/lesh/coding/dimos/.venv/bin/python3: No module named md_babel_py.session_server - +@abstractmethod +def publish(self, topic: TopicT, message: MsgT) -> None: + """Publish a message to a topic.""" + ... +@abstractmethod +def subscribe( + self, topic: TopicT, callback: Callable[[MsgT, TopicT], None] +) -> Callable[[], None]: + """Subscribe to a topic with a callback. returns unsubscribe function""" + ... ``` Key points: diff --git a/docs/development/README.md b/docs/development/README.md index 9a6f296fbc..06a6764158 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,6 +1,6 @@ # Development Guide -1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) +1. [How to set up your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) 2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (which files to edit, debugging help, etc) 3. [How to make a PR](#3-how-to-make-a-pr) (our expectations for a PR) @@ -16,12 +16,12 @@ All the setup options are for your convenience. If you can get DimOS running on ### Why pick this option? (pros/cons/when-to-use) -* Downside: mutates your global system, causing (and receiving) side effects causes it to be unreliable +* Downside: mutates your global system, which can create side effects and make it less reliable * Upside: Often good for a quick hack or exploring * Upside: Sometimes easier for CUDA/GPU acceleration * Use when: you understand system package management (arch linux user) or you don't care about making changes to your system -### How to setup DimOS +### How to set up DimOS ```bash # System dependencies @@ -230,11 +230,11 @@ This will save the rerun data to `rerun.json` in the current directory. * If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) * If you want to add a camera driver see [depth_camera_integration.md](/docs/development/depth_camera_integration.md) -* For edits to manipulation see [manipulation](/dimos/hardware/manipulators/REAME.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) +* For edits to manipulation see [manipulation](/dimos/hardware/manipulators/README.md) and the related modules under `dimos/manipulation/`. * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. -* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. +* `dimos/msgs/`: If you're trying to find a message type to send over a stream, look here. * `dimos/dashboard/`: Contains code related to visualization. * `dimos/protocol/`: Defines low level stuff for communication between modules. * See `dimos/` for the remainder @@ -256,7 +256,7 @@ pytest # run all tests at or below the current directory | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be run in CI and for cases that require hardware or visual inspection (pointcloud merging visualization, etc). You can enable a tag by selecting -m - these are configured in `./pyproject.toml` diff --git a/docs/development/depth_camera_integration.md b/docs/development/depth_camera_integration.md index 4fca10da4e..3ff162d646 100644 --- a/docs/development/depth_camera_integration.md +++ b/docs/development/depth_camera_integration.md @@ -6,7 +6,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## Add a New Depth Camera 1) **Create a new driver module** - - Path: `dimos/dimos/hardware/sensors/camera//camera.py` + - Path: `dimos/hardware/sensors/camera//camera.py` - Export a blueprint in `/__init__.py` (match the `realsense` / `zed` pattern). 2) **Define config** @@ -57,7 +57,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## TF: Required Frames and Transforms -Frame names are defined by the abstract depth camera spec (`dimos/dimos/hardware/sensors/camera/spec.py`). +Frame names are defined by the abstract depth camera spec (`dimos/hardware/sensors/camera/spec.py`). Use the properties below to ensure consistent naming: - `_camera_link`: base link for the camera module (usually `{camera_name}_link`) @@ -111,8 +111,8 @@ For `ObjectSceneRegistrationModule`, the required inputs are: - Overlay annotations and aggregated pointclouds See: -- `dimos/dimos/perception/object_scene_registration.py` -- `dimos/dimos/perception/demo_object_scene_registration.py` +- `dimos/perception/object_scene_registration.py` +- `dimos/perception/demo_object_scene_registration.py` Quick wiring example: @@ -143,5 +143,5 @@ Install Foxglove from: - **Skills** are callable methods (decorated with `@skill`) exposed by `SkillModule` for agents. Reference: -- Modules overview: `dimos/docs/concepts/modules.md` -- TF fundamentals: `dimos/docs/api/transforms.md` +- Modules overview: `/docs/concepts/modules.md` +- TF fundamentals: `/docs/api/transforms.md` diff --git a/docs/development/writing_docs/README.md b/docs/development/writing_docs/README.md index a848497e23..abced8b706 100644 --- a/docs/development/writing_docs/README.md +++ b/docs/development/writing_docs/README.md @@ -5,13 +5,13 @@ Note: as of the DimOS beta, not all existing docs conform to this guide, but new ## Need-to-know Things 1. Where to put your docs. - - Some docs are under `docs/` (like this one) but others are stored in the actual codebase, like `dimos/robots/drones/README.md`. - - If your docs have code examples, and are somewhere under `docs/` those code examples must be executable. See [codeblocks guide](/docs/development/writing_docs/code_blocks.md) for details and instructions on how to execute your code examples. + - Some docs are under `docs/` (like this one) but others are stored in the actual codebase, like `dimos/robot/drone/README.md`. + - If your docs have code examples and are somewhere under `docs/`, those code examples must be executable. See [codeblocks guide](/docs/development/writing_docs/codeblocks.md) for details and instructions on how to execute your code examples. - If your docs nicely *introduce* a new API, or they are a tutorial, then put them in `docs/concepts/` (even if they are about a specific API). - If the docs are highly technical or exhaustive there are a three options: - If your docs are about a user-facing API (ex: the reader can follow your instructions without cloning dimos) then put them in `docs/api/`. - Otherwise (if the reader is modifying their own copy of the dimos codebase) then your docs have two options: - 1. You can choose to store your docs next to relevant python files (ex: `dimos/robots/drones/README.md`), and we are less strict about the contents (code examples don't need to be executable) **BUT**, you need edit something in `docs/development/` or `docs/api/` that to add a reference/link to those docs (don't create "dangling" documentation). + 1. You can choose to store your docs next to relevant python files (ex: `dimos/robot/drone/README.md`), and we are less strict about the contents (code examples don't need to be executable) **BUT**, you need to edit something in `docs/development/` or `docs/api/` to add a reference/link to those docs (don't create "dangling" documentation). 2. Alternatively, you can put your docs in `docs/development/`. Code examples there should be executable. 2. Even if you know how to link to other docs, read our [how we do doc linking guide](/docs/development/writing_docs/doclinks.md). 3. Even if you know how to create diagrams on your own, read our [how we do diagrams guide](/docs/development/writing_docs/diagram_practices.md). diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index ef4eb0ce8a..8c6264f230 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -1,4 +1,4 @@ -We have many diagraming tools. View source code of this page to see examples. +We have many diagramming tools. View source code of this page to see examples. # How to make diagrams From 530946796551c07c91aecc4869188394d5d1c952 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 21:54:17 -0800 Subject: [PATCH 122/136] make docs consistent --- docs/concepts/modules.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index 6f167cb43e..344efa0774 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -47,8 +47,8 @@ print(CameraModule.io()) ├─ color_image: Image ├─ camera_info: CameraInfo │ - ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool ├─ RPC start() + ├─ RPC stop() │ ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) ``` @@ -58,7 +58,7 @@ We can see that the camera module outputs two streams: - `color_image` with [sensor_msgs.Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) type - `camera_info` with [sensor_msgs.CameraInfo](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) type -It offers one module-specific RPC call: `start()`. You will also see the base `set_transport(...)` RPC from `Module`. The module still has a `stop()` lifecycle method, but it's called on the local instance (or by the coordinator), not via RPC. +It offers two RPC calls: `start()` and `stop()` (lifecycle methods). It also exposes an agentic [skill](/docs/concepts/blueprints.md#defining-skills) called `video_stream` (more on skills in the Blueprints guide). From d42049369fadad90b16eb9ad0e5b61bcfd00d394 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 21:55:56 -0800 Subject: [PATCH 123/136] add back lcmspy (accidental removal) --- docs/concepts/transports.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/concepts/transports.md b/docs/concepts/transports.md index 6f37f8c631..6c9e5fca99 100644 --- a/docs/concepts/transports.md +++ b/docs/concepts/transports.md @@ -138,6 +138,12 @@ lcm.stop() Received velocity: x=1.0, y=0.0, z=0.5 ``` +### Inspecting LCM traffic (CLI) + +- `dimos lcmspy` shows topic frequency/bandwidth stats. +- `dimos topic echo /topic` listens on typed channels like `/topic#pkg.Msg` and decodes automatically. +- `dimos topic echo /topic TypeName` is the explicit legacy form. + ## Encoder Mixins Transports can use encoder mixins to serialize messages. The `PubSubEncoderMixin` pattern wraps publish/subscribe to encode/decode automatically: From b281ae7aaf06bd7d1d1146fbeb059792b2b022f2 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Mon, 19 Jan 2026 22:02:54 -0800 Subject: [PATCH 124/136] better wording --- docs/development/writing_docs/diagram_practices.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index 8c6264f230..81f3da73a5 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -2,8 +2,8 @@ We have many diagramming tools. View source code of this page to see examples. # How to make diagrams -1. Use Pikchr (language) to define a diagram -2. Use `md-babel-py` from [codeblocks.md](/docs/development/writing_docs/codeblocks.md) to execute the diagram and insert the result. +1. First define a diagram using a codeblock (examples below). See [Pikchr](https://pikchr.org/) for more details on syntax. +2. Then use the cli tool `md-babel-py` (ex: `md-babel-py run README.md`) to generate the diagram. See [codeblocks.md](/docs/development/writing_docs/codeblocks.md) for how to get the `md-babel-py` cli tool. # Pikchr From c20128f3089f5b56b2ca728b0a259eba923054cb Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 12:45:09 -0800 Subject: [PATCH 125/136] improve REPO_ROOT --- bin/doclinks | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/doclinks b/bin/doclinks index c2b05e8acb..5dee1c69b0 100755 --- a/bin/doclinks +++ b/bin/doclinks @@ -1,3 +1,3 @@ #!/usr/bin/env bash -REPO_ROOT=$(git rev-parse --show-toplevel) +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" From 483bce73ce2abb08eb622620ed43817f8e119bb2 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 12:45:19 -0800 Subject: [PATCH 126/136] simplify --- docs/concepts/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/README.md b/docs/concepts/README.md index ed5c00ab01..ff1e284d78 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -1,6 +1,6 @@ # Concepts -If you are trying to learn a new API or know what tools you should use, then you are in the right place. In contrast, if you already know the API you want to use and just need details on how to use it, see [the API reference](/docs/api/README.md). +This page explains general concepts. For specific API docs see [the API reference](/docs/api/README.md). ## Table of Contents From 4faba9dd813f6ceef207ddd6dea6c8347cf23ac1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 12:47:28 -0800 Subject: [PATCH 127/136] wording --- README.md | 2 +- docs/concepts/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9ccc7de197..9f8ee5f59e 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Core Features: test dimensional applications in real-time. Control your robot via waypoint, agent query, keyboard, VR, more. - **Modules:** Standalone components (equivalent to ROS nodes) that publish and subscribe to typed - In/Out streams that communicate over DimOS transports. The building blocks of Dimensional. + In/Out streams that communicate over DimOS transports. The primary components of Dimensional. - **Agents (experimental):** DimOS agents understand physical space, subscribe to sensor streams, and call **physical** tools. Emergence appears when agents have physical agency. - **MCP (experimental):** Vibecode robots by giving your AI editor (Cursor, Claude Code) MCP access to run diff --git a/docs/concepts/README.md b/docs/concepts/README.md index ff1e284d78..0f82783099 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -4,7 +4,7 @@ This page explains general concepts. For specific API docs see [the API referenc ## Table of Contents -- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes. +- [Modules](/docs/concepts/modules.md): The primary units of deployment in DimOS, modules run in parallel and are python classes. - [Streams](/docs/api/sensor_streams/README.md): How modules communicate, a Pub / Sub system. - [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. - [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). From 40ecd57044f8d1d8efc1d22545786328b80f21a1 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 13:01:45 -0800 Subject: [PATCH 128/136] should be fully runnable in theory, test next --- docs/concepts/blueprints.md | 97 +++++++++++++++++++++++-------------- 1 file changed, 61 insertions(+), 36 deletions(-) diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 6caa1f3004..948ffbc0f2 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -6,31 +6,26 @@ You don't typically want to run a single module, so multiple blueprints are hand You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: -```python +```python session=blueprint-ex1 from dimos.core.blueprints import create_module_blueprint from dimos.core import Module, rpc class ConnectionModule(Module): - @rpc - def start(): - pass - - @rpc - def stop(): - pass + def __init__(self, arg1, arg2, kwarg='value') -> None: + super().__init__() blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') ``` But the same thing can be accomplished more succinctly as: -```python +```python session=blueprint-ex1 connection = ConnectionModule.blueprint ``` Now you can create the blueprint with: -```python +```python session=blueprint-ex1 blueprint = connection('arg1', 'arg2', kwarg='value') ``` @@ -38,9 +33,23 @@ blueprint = connection('arg1', 'arg2', kwarg='value') You can link multiple blueprints together with `autoconnect`: -```python +```python session=blueprint-ex1 from dimos.core.blueprints import autoconnect +class Module1(Module): + def __init__(self, arg1) -> None: + super().__init__() + +class Module2(Module): + ... + +class Module3(Module): + ... + +module1 = Module1.blueprint +module2 = Module2.blueprint +module3 = Module3.blueprint + blueprint = autoconnect( module1(), module2(), @@ -50,7 +59,16 @@ blueprint = autoconnect( `blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: -```python +```python session=blueprint-ex1 +class Module4(Module): + ... + +class Module5(Module): + ... + +module4 = Module4.blueprint +module5 = Module5.blueprint + expanded_blueprint = autoconnect( blueprint, module4(), @@ -64,11 +82,11 @@ Blueprints are frozen data classes, and `autoconnect()` always constructs an exp If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: -```python +```python session=blueprint-ex1 blueprint = autoconnect( - module_a(arg1=1), - module_b(), - module_a(arg1=2), # This one is used, the first is discarded + module1(arg1=1), + module2(), + module1(arg1=2), # This one is used, the first is discarded ) ``` @@ -78,7 +96,7 @@ This is so you can "inherit" from one blueprint but override something you need Imagine you have this code: -```python +```python session=blueprint-ex1 from functools import partial from dimos.core.blueprints import create_module_blueprint, autoconnect @@ -114,10 +132,17 @@ By default `LCMTransport` is used if the object supports `lcm_encode`. If it doe You can override transports with the `transports` method. It returns a new blueprint in which the override is set. -```python -blueprint = autoconnect(...) -expanded_blueprint = autoconnect(blueprint, ...) -blueprint = blueprint.transports({ +```python session=blueprint-ex1 +base_blueprint = autoconnect( + module1(arg1=1), + module2(), +) +expanded_blueprint = autoconnect( + base_blueprint, + module4(), + module5(), +) +base_blueprint = base_blueprint.transports({ ("image", Image): pSHMTransport( "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE ), @@ -125,13 +150,13 @@ blueprint = blueprint.transports({ }) ``` -Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `base_blueprint`, not the second. ## Remapping connections Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: -```python +```python session=blueprint-ex2 from dimos.core.blueprints import autoconnect from dimos.core import Module, rpc @@ -161,7 +186,7 @@ After remapping: If you want to override the topic, you still have to do it manually: -```python +```python session=blueprint-ex2 blueprint .remappings([ (ConnectionModule, 'color_image', 'rgb_image'), @@ -175,7 +200,7 @@ blueprint Each module can optionally take a `global_config` option in `__init__`. E.g.: -```python +```python session=blueprint-ex3 from dimos.core import Module, rpc class ModuleA(Module): @@ -186,7 +211,7 @@ class ModuleA(Module): The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: -```python +```python session=blueprint-ex3 blueprint = blueprint.global_config(n_dask_workers=8) ``` @@ -194,7 +219,7 @@ blueprint = blueprint.global_config(n_dask_workers=8) Imagine you have this code: -```python +```python session=blueprint-ex3 from dimos.core import Module, rpc class ModuleA(Module): @@ -212,7 +237,7 @@ And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. -```python +```python session=blueprint-ex3 from dimos.core import Module, rpc class ModuleB(Module): @@ -227,7 +252,7 @@ class ModuleB(Module): You can also request multiple methods at a time: -```python +```python session=blueprint-ex3 method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") ``` @@ -237,7 +262,7 @@ There is an alternative way of receiving RPC methods. It is useful when you want You can use it by defining a method like `set__`: -```python +```python session=blueprint-ex3 from dimos.core import Module, rpc class ModuleB(Module): @@ -258,7 +283,7 @@ In the previous examples, you can only call methods in a module called `ModuleA` You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. -```python +```python session=blueprint-ex3 from abc import ABC, abstractmethod from dimos.core import Module, rpc @@ -288,7 +313,7 @@ class ModuleB(Module): The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: -```python +```python session=blueprint-ex3 blueprint = autoconnect( ProperTime.blueprint(), # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time @@ -302,7 +327,7 @@ If both are deployed, the blueprint will throw an error because it's ambiguous. Skills have to be registered with `AgentSpec.register_skills(self)`. -```python +```python session=blueprint-ex4 from dimos.core import Module, rpc from dimos.core.skill_module import SkillModule from dimos.protocol.skill.skill import skill @@ -328,7 +353,7 @@ class SomeSkill(Module): Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: -```python +```python session=blueprint-ex4 from dimos.core.skill_module import SkillModule from dimos.protocol.skill.skill import skill @@ -343,7 +368,7 @@ class SomeSkill(SkillModule): All you have to do to build a blueprint is call: -```python +```python session=blueprint-ex4 module_coordinator = blueprint.build(global_config=config) ``` @@ -353,7 +378,7 @@ This returns a `ModuleCoordinator` instance that manages all deployed modules. You can block the thread until it exits with: -```python +```python session=blueprint-ex4 module_coordinator.loop() ``` From e3548a88e7a07be6f66250a3a0ff5986f323a66d Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 19:02:09 -0800 Subject: [PATCH 129/136] add doclinks command --- bin/doclinks | 3 --- docs/development/writing_docs/doclinks.md | 22 ++++++++++++---------- pyproject.toml | 1 + 3 files changed, 13 insertions(+), 13 deletions(-) delete mode 100755 bin/doclinks diff --git a/bin/doclinks b/bin/doclinks deleted file mode 100755 index 5dee1c69b0..0000000000 --- a/bin/doclinks +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env bash -REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md index e9a6ee9d8c..9d76b9d87f 100644 --- a/docs/development/writing_docs/doclinks.md +++ b/docs/development/writing_docs/doclinks.md @@ -12,10 +12,12 @@ ## Usage +After installing the `dimos` package, the `doclinks` command should be available. + ```bash -bin/doclinks docs/guide.md # single file -bin/doclinks docs/ # directory -bin/doclinks --dry-run ... # preview only +doclinks docs/guide.md # single file +doclinks docs/ # directory +doclinks --dry-run ... # preview only ``` ## Full Documentation @@ -60,26 +62,26 @@ See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. ```bash # Process a single file -bin/doclinks docs/guide.md +doclinks docs/guide.md # Process a directory recursively -bin/doclinks docs/ +doclinks docs/ # Relative links (from doc location) -bin/doclinks --link-mode relative docs/ +doclinks --link-mode relative docs/ # GitHub links -bin/doclinks --link-mode github \ +doclinks --link-mode github \ --github-url https://github.com/org/repo docs/ # Dry run (preview changes) -bin/doclinks --dry-run docs/ +doclinks --dry-run docs/ # CI check (exit 1 if changes needed) -bin/doclinks --check docs/ +doclinks --check docs/ # Watch mode (auto-update on changes) -bin/doclinks --watch docs/ +doclinks --watch docs/ ``` ## Options diff --git a/pyproject.toml b/pyproject.toml index 5c824bdc16..afe868a091 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,6 +77,7 @@ foxglove-bridge = "dimos.utils.cli.foxglove_bridge.run_foxglove_bridge:main" skillspy = "dimos.utils.cli.skillspy.skillspy:main" agentspy = "dimos.utils.cli.agentspy.agentspy:main" humancli = "dimos.utils.cli.human.humanclianim:main" +doclinks = "dimos.utils.docs.doclinks:main" dimos = "dimos.robot.cli.dimos:main" [project.optional-dependencies] From d7fd814647816e2dfa416e9f64fbbd39fb9bc5a6 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 19:02:49 -0800 Subject: [PATCH 130/136] Revert "add doclinks command" This reverts commit e3548a88e7a07be6f66250a3a0ff5986f323a66d. --- bin/doclinks | 3 +++ docs/development/writing_docs/doclinks.md | 22 ++++++++++------------ pyproject.toml | 1 - 3 files changed, 13 insertions(+), 13 deletions(-) create mode 100755 bin/doclinks diff --git a/bin/doclinks b/bin/doclinks new file mode 100755 index 0000000000..5dee1c69b0 --- /dev/null +++ b/bin/doclinks @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md index 9d76b9d87f..e9a6ee9d8c 100644 --- a/docs/development/writing_docs/doclinks.md +++ b/docs/development/writing_docs/doclinks.md @@ -12,12 +12,10 @@ ## Usage -After installing the `dimos` package, the `doclinks` command should be available. - ```bash -doclinks docs/guide.md # single file -doclinks docs/ # directory -doclinks --dry-run ... # preview only +bin/doclinks docs/guide.md # single file +bin/doclinks docs/ # directory +bin/doclinks --dry-run ... # preview only ``` ## Full Documentation @@ -62,26 +60,26 @@ See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. ```bash # Process a single file -doclinks docs/guide.md +bin/doclinks docs/guide.md # Process a directory recursively -doclinks docs/ +bin/doclinks docs/ # Relative links (from doc location) -doclinks --link-mode relative docs/ +bin/doclinks --link-mode relative docs/ # GitHub links -doclinks --link-mode github \ +bin/doclinks --link-mode github \ --github-url https://github.com/org/repo docs/ # Dry run (preview changes) -doclinks --dry-run docs/ +bin/doclinks --dry-run docs/ # CI check (exit 1 if changes needed) -doclinks --check docs/ +bin/doclinks --check docs/ # Watch mode (auto-update on changes) -doclinks --watch docs/ +bin/doclinks --watch docs/ ``` ## Options diff --git a/pyproject.toml b/pyproject.toml index afe868a091..5c824bdc16 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,7 +77,6 @@ foxglove-bridge = "dimos.utils.cli.foxglove_bridge.run_foxglove_bridge:main" skillspy = "dimos.utils.cli.skillspy.skillspy:main" agentspy = "dimos.utils.cli.agentspy.agentspy:main" humancli = "dimos.utils.cli.human.humanclianim:main" -doclinks = "dimos.utils.docs.doclinks:main" dimos = "dimos.robot.cli.dimos:main" [project.optional-dependencies] From ae63769f3d29f83f4a5c4e7900957f79be20fcf4 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 19:58:26 -0800 Subject: [PATCH 131/136] improve syntax --- docs/development/writing_docs/codeblocks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/writing_docs/codeblocks.md b/docs/development/writing_docs/codeblocks.md index 5b5b8a4dfc..a928f1ceb4 100644 --- a/docs/development/writing_docs/codeblocks.md +++ b/docs/development/writing_docs/codeblocks.md @@ -30,7 +30,7 @@ nix run . -- run README.md --stdout docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout # Or build locally via Nix -nix build .#docker # builds tarball to ./result +nix build '.#docker' # builds tarball to ./result docker load < result # loads image from tarball docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout ``` From 96554bdb5a4c3990bd8110ed4e500c0de4242861 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 20:18:54 -0800 Subject: [PATCH 132/136] make blueprints.md executable (tested) --- docs/concepts/blueprints.md | 39 +++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 948ffbc0f2..686aff60d5 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -100,15 +100,16 @@ Imagine you have this code: from functools import partial from dimos.core.blueprints import create_module_blueprint, autoconnect -from dimos.core import Module, rpc +from dimos.core import Module, rpc, Out, In +from dimos.msgs.sensor_msgs import Image class ModuleA(Module): image: Out[Image] - start_explore: Out[Bool] + start_explore: Out[bool] class ModuleB(Module): image: In[Image] - begin_explore: In[Bool] + begin_explore: In[bool] module_a = partial(create_module_blueprint, ModuleA) module_b = partial(create_module_blueprint, ModuleB) @@ -133,6 +134,8 @@ By default `LCMTransport` is used if the object supports `lcm_encode`. If it doe You can override transports with the `transports` method. It returns a new blueprint in which the override is set. ```python session=blueprint-ex1 +from dimos.core.transport import pSHMTransport, pLCMTransport + base_blueprint = autoconnect( module1(arg1=1), module2(), @@ -144,9 +147,9 @@ expanded_blueprint = autoconnect( ) base_blueprint = base_blueprint.transports({ ("image", Image): pSHMTransport( - "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE + "/go2/color_image", default_capacity=1920 * 1080 * 3, # 1920x1080 frame x 3 (RGB) x uint8 ), - ("start_explore", Bool): pLCMTransport(), + ("start_explore", bool): pLCMTransport("/start_explore"), }) ``` @@ -158,7 +161,8 @@ Sometimes you need to rename a connection to match what other modules expect. Yo ```python session=blueprint-ex2 from dimos.core.blueprints import autoconnect -from dimos.core import Module, rpc +from dimos.core import Module, rpc, Out, In +from dimos.msgs.sensor_msgs import Image class ConnectionModule(Module): color_image: Out[Image] # Outputs on 'color_image' @@ -187,11 +191,10 @@ After remapping: If you want to override the topic, you still have to do it manually: ```python session=blueprint-ex2 -blueprint -.remappings([ +from dimos.core.transport import LCMTransport +blueprint.remappings([ (ConnectionModule, 'color_image', 'rgb_image'), -]) -.transports({ +]).transports({ ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), }) ``` @@ -202,6 +205,7 @@ Each module can optionally take a `global_config` option in `__init__`. E.g.: ```python session=blueprint-ex3 from dimos.core import Module, rpc +from dimos.core.global_config import GlobalConfig class ModuleA(Module): @@ -212,7 +216,7 @@ class ModuleA(Module): The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: ```python session=blueprint-ex3 -blueprint = blueprint.global_config(n_dask_workers=8) +blueprint = ModuleA.blueprint().global_config(n_dask_workers=8) ``` ## Calling the methods of other modules @@ -253,7 +257,9 @@ class ModuleB(Module): You can also request multiple methods at a time: ```python session=blueprint-ex3 -method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +class ModuleB(Module): + def request_the_time(self) -> None: + method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") ``` ## Alternative RPC calls @@ -264,6 +270,7 @@ You can use it by defining a method like `set__`: ```python session=blueprint-ex3 from dimos.core import Module, rpc +from dimos.core.rpc_client import RpcCall class ModuleB(Module): @rpc # Note that it has to be an rpc method. @@ -285,14 +292,14 @@ You can do so by extracting the common interface as an `ABC` (abstract base clas ```python session=blueprint-ex3 from abc import ABC, abstractmethod - +from dimos.core.blueprints import autoconnect from dimos.core import Module, rpc class TimeInterface(ABC): @abstractmethod def get_time(self): ... -class ProperTime(TimeInterface): +class ProperTime(Module, TimeInterface): def get_time(self): return "13:00" @@ -331,6 +338,8 @@ Skills have to be registered with `AgentSpec.register_skills(self)`. from dimos.core import Module, rpc from dimos.core.skill_module import SkillModule from dimos.protocol.skill.skill import skill +from dimos.core.rpc_client import RpcCall +from dimos.core.global_config import GlobalConfig class SomeSkill(Module): @@ -369,7 +378,7 @@ class SomeSkill(SkillModule): All you have to do to build a blueprint is call: ```python session=blueprint-ex4 -module_coordinator = blueprint.build(global_config=config) +module_coordinator = SomeSkill.blueprint().build(global_config=GlobalConfig()) ``` This returns a `ModuleCoordinator` instance that manages all deployed modules. From de3f34da3dc84a8454c2b1a512ba78ccbb616288 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 20:31:08 -0800 Subject: [PATCH 133/136] neutralize edits --- README.md | 2 +- bin/doclinks | 3 - docs/api/README.md | 11 - docs/api/configuration.md | 2 +- docs/api/sensor_streams/quality_filter.md | 2 +- docs/api/sensor_streams/storage_replay.md | 2 +- docs/api/sensor_streams/temporal_alignment.md | 4 +- docs/api/visualization.md | 2 +- docs/concepts/README.md | 12 - docs/concepts/blueprints.md | 159 +++------- docs/concepts/modules.md | 13 +- docs/concepts/transports.md | 14 +- docs/development/README.md | 22 +- docs/development/depth_camera_integration.md | 12 +- docs/development/dimos_run.md | 20 +- docs/development/writing_docs/README.md | 17 -- .../writing_docs/assets/pikchr_basic.svg | 12 - .../writing_docs/assets/pikchr_sizing.svg | 13 - docs/development/writing_docs/codeblocks.md | 289 ++++++++++++------ .../writing_docs/diagram_practices.md | 58 +++- docs/development/writing_docs/doclinks.md | 41 +-- 21 files changed, 346 insertions(+), 364 deletions(-) delete mode 100755 bin/doclinks delete mode 100644 docs/api/README.md delete mode 100644 docs/concepts/README.md delete mode 100644 docs/development/writing_docs/README.md delete mode 100644 docs/development/writing_docs/assets/pikchr_basic.svg delete mode 100644 docs/development/writing_docs/assets/pikchr_sizing.svg diff --git a/README.md b/README.md index 9f8ee5f59e..9ccc7de197 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Core Features: test dimensional applications in real-time. Control your robot via waypoint, agent query, keyboard, VR, more. - **Modules:** Standalone components (equivalent to ROS nodes) that publish and subscribe to typed - In/Out streams that communicate over DimOS transports. The primary components of Dimensional. + In/Out streams that communicate over DimOS transports. The building blocks of Dimensional. - **Agents (experimental):** DimOS agents understand physical space, subscribe to sensor streams, and call **physical** tools. Emergence appears when agents have physical agency. - **MCP (experimental):** Vibecode robots by giving your AI editor (Cursor, Claude Code) MCP access to run diff --git a/bin/doclinks b/bin/doclinks deleted file mode 100755 index 5dee1c69b0..0000000000 --- a/bin/doclinks +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env bash -REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" diff --git a/docs/api/README.md b/docs/api/README.md deleted file mode 100644 index 1a14f83640..0000000000 --- a/docs/api/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# API - -Note: Please see [Concepts](/docs/concepts/README.md) before diving into these API docs. These docs are designed to be technical, cover specific tooling, and give examples of using those tools (argument types) rather than explaining when to use the tool. - - -# Table of Contents - -- [adding configs to your modules](./configuration.md) -- [controlling how visualizations are rendered](./visualization.md) -- [understanding sensor streams](./sensor_streams/README.md) -- [creating coordinate frames and transforms for your hardware](./transforms.md) diff --git a/docs/api/configuration.md b/docs/api/configuration.md index 162c343ffa..a9c8de0268 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules](/docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame IDs, etc. +[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids, etc. ```python from dataclasses import dataclass diff --git a/docs/api/sensor_streams/quality_filter.md b/docs/api/sensor_streams/quality_filter.md index 01d3b9367d..26d40733fd 100644 --- a/docs/api/sensor_streams/quality_filter.md +++ b/docs/api/sensor_streams/quality_filter.md @@ -50,7 +50,7 @@ Qualities: [0.9] For camera streams, we provide `sharpness_barrier` which uses the image's sharpness score. -Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Storage & Replay](/docs/api/sensor_streams/storage_replay.md) toolkit, which provides access to recorded robot data: +Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Replay](/docs/old/testing_stream_reply.md) toolkit, which provides access to recorded robot data: ```python session=qb from dimos.utils.testing import TimedSensorReplay diff --git a/docs/api/sensor_streams/storage_replay.md b/docs/api/sensor_streams/storage_replay.md index 1a31591736..66e913b197 100644 --- a/docs/api/sensor_streams/storage_replay.md +++ b/docs/api/sensor_streams/storage_replay.md @@ -202,7 +202,7 @@ Each pickle file contains a tuple `(timestamp, data)`: Files are numbered sequentially: `000.pickle`, `001.pickle`, etc. -Recordings are stored in the `data/` directory. See [Data Loading](/docs/development/large_file_management.md) for how data storage works, including Git LFS handling for large datasets. +Recordings are stored in the `data/` directory. See [Data Loading](/docs/data.md) for how data storage works, including Git LFS handling for large datasets. ## API Reference diff --git a/docs/api/sensor_streams/temporal_alignment.md b/docs/api/sensor_streams/temporal_alignment.md index 66230c9d54..b552ac54cc 100644 --- a/docs/api/sensor_streams/temporal_alignment.md +++ b/docs/api/sensor_streams/temporal_alignment.md @@ -34,7 +34,7 @@ Below we set up replay of real camera and lidar data from the Unitree Go2 robot.
Stream Setup -You can read more about [sensor storage here](storage_replay.md) and [LFS data storage here](/docs/development/large_file_management.md). +You can read more about [sensor storage here](storage_replay.md) and [LFS data store here](/docs/data.md). ```python session=align no-result from reactivex import Subject @@ -70,7 +70,7 @@ lidar_stream = lidar_replay.stream(from_timestamp=seek_ts, duration=2.0).pipe(
-Streams would normally come from an actual robot into your module via `In` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. +Streams would normally come from an actual robot into your module via `IN` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. Assume we have them. Let's align them. diff --git a/docs/api/visualization.md b/docs/api/visualization.md index 08919b7a73..51fa20d655 100644 --- a/docs/api/visualization.md +++ b/docs/api/visualization.md @@ -54,7 +54,7 @@ VIEWER_BACKEND=foxglove dimos run unitree-go2 - Foxglove bridge on ws://localhost:8765 - No Rerun (saves resources) - Better performance with larger maps/higher resolution -- Open layout: `assets/foxglove_dashboards/old/foxglove_unitree_lcm_dashboard.json` +- Open layout: `dimos/assets/foxglove_dashboards/go2.json` --- diff --git a/docs/concepts/README.md b/docs/concepts/README.md deleted file mode 100644 index 0f82783099..0000000000 --- a/docs/concepts/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Concepts - -This page explains general concepts. For specific API docs see [the API reference](/docs/api/README.md). - -## Table of Contents - -- [Modules](/docs/concepts/modules.md): The primary units of deployment in DimOS, modules run in parallel and are python classes. -- [Streams](/docs/api/sensor_streams/README.md): How modules communicate, a Pub / Sub system. -- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. -- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). -- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). -- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 686aff60d5..0a3e2ceaf5 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -6,26 +6,19 @@ You don't typically want to run a single module, so multiple blueprints are hand You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: -```python session=blueprint-ex1 -from dimos.core.blueprints import create_module_blueprint -from dimos.core import Module, rpc - -class ConnectionModule(Module): - def __init__(self, arg1, arg2, kwarg='value') -> None: - super().__init__() - +```python blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') ``` -But the same thing can be accomplished more succinctly as: +But the same thing can be acomplished more succinctly as: -```python session=blueprint-ex1 +```python connection = ConnectionModule.blueprint ``` Now you can create the blueprint with: -```python session=blueprint-ex1 +```python blueprint = connection('arg1', 'arg2', kwarg='value') ``` @@ -33,23 +26,7 @@ blueprint = connection('arg1', 'arg2', kwarg='value') You can link multiple blueprints together with `autoconnect`: -```python session=blueprint-ex1 -from dimos.core.blueprints import autoconnect - -class Module1(Module): - def __init__(self, arg1) -> None: - super().__init__() - -class Module2(Module): - ... - -class Module3(Module): - ... - -module1 = Module1.blueprint -module2 = Module2.blueprint -module3 = Module3.blueprint - +```python blueprint = autoconnect( module1(), module2(), @@ -59,16 +36,7 @@ blueprint = autoconnect( `blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: -```python session=blueprint-ex1 -class Module4(Module): - ... - -class Module5(Module): - ... - -module4 = Module4.blueprint -module5 = Module5.blueprint - +```python expanded_blueprint = autoconnect( blueprint, module4(), @@ -82,11 +50,11 @@ Blueprints are frozen data classes, and `autoconnect()` always constructs an exp If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: -```python session=blueprint-ex1 +```python blueprint = autoconnect( - module1(arg1=1), - module2(), - module1(arg1=2), # This one is used, the first is discarded + module_a(arg1=1), + module_b(), + module_a(arg1=2), # This one is used, the first is discarded ) ``` @@ -96,20 +64,14 @@ This is so you can "inherit" from one blueprint but override something you need Imagine you have this code: -```python session=blueprint-ex1 -from functools import partial - -from dimos.core.blueprints import create_module_blueprint, autoconnect -from dimos.core import Module, rpc, Out, In -from dimos.msgs.sensor_msgs import Image - +```python class ModuleA(Module): image: Out[Image] - start_explore: Out[bool] + start_explore: Out[Bool] class ModuleB(Module): image: In[Image] - begin_explore: In[bool] + begin_explore: In[Bool] module_a = partial(create_module_blueprint, ModuleA) module_b = partial(create_module_blueprint, ModuleB) @@ -133,37 +95,24 @@ By default `LCMTransport` is used if the object supports `lcm_encode`. If it doe You can override transports with the `transports` method. It returns a new blueprint in which the override is set. -```python session=blueprint-ex1 -from dimos.core.transport import pSHMTransport, pLCMTransport - -base_blueprint = autoconnect( - module1(arg1=1), - module2(), -) -expanded_blueprint = autoconnect( - base_blueprint, - module4(), - module5(), -) -base_blueprint = base_blueprint.transports({ +```python +blueprint = autoconnect(...) +expanded_blueprint = autoconnect(blueprint, ...) +blueprint = blueprint.transports({ ("image", Image): pSHMTransport( - "/go2/color_image", default_capacity=1920 * 1080 * 3, # 1920x1080 frame x 3 (RGB) x uint8 + "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE ), - ("start_explore", bool): pLCMTransport("/start_explore"), + ("start_explore", Bool): pLCMTransport(), }) ``` -Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `base_blueprint`, not the second. +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. ## Remapping connections Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: -```python session=blueprint-ex2 -from dimos.core.blueprints import autoconnect -from dimos.core import Module, rpc, Out, In -from dimos.msgs.sensor_msgs import Image - +```python class ConnectionModule(Module): color_image: Out[Image] # Outputs on 'color_image' @@ -190,11 +139,12 @@ After remapping: If you want to override the topic, you still have to do it manually: -```python session=blueprint-ex2 -from dimos.core.transport import LCMTransport -blueprint.remappings([ +```python +blueprint +.remappings([ (ConnectionModule, 'color_image', 'rgb_image'), -]).transports({ +]) +.transports({ ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), }) ``` @@ -203,10 +153,7 @@ blueprint.remappings([ Each module can optionally take a `global_config` option in `__init__`. E.g.: -```python session=blueprint-ex3 -from dimos.core import Module, rpc -from dimos.core.global_config import GlobalConfig - +```python class ModuleA(Module): def __init__(self, global_config: GlobalConfig | None = None): @@ -215,17 +162,15 @@ class ModuleA(Module): The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: -```python session=blueprint-ex3 -blueprint = ModuleA.blueprint().global_config(n_dask_workers=8) +```python +blueprint = blueprint.global_config(n_dask_workers=8) ``` ## Calling the methods of other modules Imagine you have this code: -```python session=blueprint-ex3 -from dimos.core import Module, rpc - +```python class ModuleA(Module): @rpc @@ -241,9 +186,7 @@ And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. -```python session=blueprint-ex3 -from dimos.core import Module, rpc - +```python class ModuleB(Module): rpc_calls: list[str] = [ "ModuleA.get_time", @@ -256,10 +199,8 @@ class ModuleB(Module): You can also request multiple methods at a time: -```python session=blueprint-ex3 -class ModuleB(Module): - def request_the_time(self) -> None: - method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +```python +method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") ``` ## Alternative RPC calls @@ -268,10 +209,7 @@ There is an alternative way of receiving RPC methods. It is useful when you want You can use it by defining a method like `set__`: -```python session=blueprint-ex3 -from dimos.core import Module, rpc -from dimos.core.rpc_client import RpcCall - +```python class ModuleB(Module): @rpc # Note that it has to be an rpc method. def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: @@ -290,16 +228,12 @@ In the previous examples, you can only call methods in a module called `ModuleA` You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. -```python session=blueprint-ex3 -from abc import ABC, abstractmethod -from dimos.core.blueprints import autoconnect -from dimos.core import Module, rpc - +```python class TimeInterface(ABC): @abstractmethod def get_time(self): ... -class ProperTime(Module, TimeInterface): +class ProperTime(TimeInterface): def get_time(self): return "13:00" @@ -320,7 +254,7 @@ class ModuleB(Module): The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: -```python session=blueprint-ex3 +```python blueprint = autoconnect( ProperTime.blueprint(), # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time @@ -334,13 +268,7 @@ If both are deployed, the blueprint will throw an error because it's ambiguous. Skills have to be registered with `AgentSpec.register_skills(self)`. -```python session=blueprint-ex4 -from dimos.core import Module, rpc -from dimos.core.skill_module import SkillModule -from dimos.protocol.skill.skill import skill -from dimos.core.rpc_client import RpcCall -from dimos.core.global_config import GlobalConfig - +```python class SomeSkill(Module): @skill @@ -362,10 +290,7 @@ class SomeSkill(Module): Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: -```python session=blueprint-ex4 -from dimos.core.skill_module import SkillModule -from dimos.protocol.skill.skill import skill - +```python class SomeSkill(SkillModule): @skill @@ -377,8 +302,8 @@ class SomeSkill(SkillModule): All you have to do to build a blueprint is call: -```python session=blueprint-ex4 -module_coordinator = SomeSkill.blueprint().build(global_config=GlobalConfig()) +```python +module_coordinator = blueprint.build(global_config=config) ``` This returns a `ModuleCoordinator` instance that manages all deployed modules. @@ -387,7 +312,7 @@ This returns a `ModuleCoordinator` instance that manages all deployed modules. You can block the thread until it exits with: -```python session=blueprint-ex4 +```python module_coordinator.loop() ``` diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index 344efa0774..ee7fbaf2c9 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -1,5 +1,5 @@ -# DimOS Modules +# Dimos Modules Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages. @@ -47,8 +47,8 @@ print(CameraModule.io()) ├─ color_image: Image ├─ camera_info: CameraInfo │ + ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool ├─ RPC start() - ├─ RPC stop() │ ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) ``` @@ -58,9 +58,9 @@ We can see that the camera module outputs two streams: - `color_image` with [sensor_msgs.Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) type - `camera_info` with [sensor_msgs.CameraInfo](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) type -It offers two RPC calls: `start()` and `stop()` (lifecycle methods). +It offers two RPC calls: `start()` and `stop()`. -It also exposes an agentic [skill](/docs/concepts/blueprints.md#defining-skills) called `video_stream` (more on skills in the Blueprints guide). +As well as an agentic [Skill](skills.md) called `video_stream` (more about this later, in [Skills Tutorial](skills.md)). We can start this module and explore the output of its streams in real time (this will use your webcam). @@ -120,7 +120,7 @@ print(Detection2DModule.io()) ├─ RPC stop() -> None ``` - +TODO: add easy way to print config Looks like the detector just needs an image input and outputs some sort of detection and annotation messages. Let's connect it to a camera. @@ -174,6 +174,3 @@ to_svg(agentic, "assets/go2_agentic.svg") ![output](assets/go2_agentic.svg) - - -To see more information on how to use Blueprints, see [Blueprints](/docs/concepts/blueprints.md). diff --git a/docs/concepts/transports.md b/docs/concepts/transports.md index 6c9e5fca99..62279b6baf 100644 --- a/docs/concepts/transports.md +++ b/docs/concepts/transports.md @@ -18,17 +18,11 @@ print(inspect.getsource(PubSub.publish)) print(inspect.getsource(PubSub.subscribe)) ``` + ``` -@abstractmethod -def publish(self, topic: TopicT, message: MsgT) -> None: - """Publish a message to a topic.""" - ... -@abstractmethod -def subscribe( - self, topic: TopicT, callback: Callable[[MsgT, TopicT], None] -) -> Callable[[], None]: - """Subscribe to a topic with a callback. returns unsubscribe function""" - ... +Session process exited unexpectedly: +/home/lesh/coding/dimos/.venv/bin/python3: No module named md_babel_py.session_server + ``` Key points: diff --git a/docs/development/README.md b/docs/development/README.md index 06a6764158..2b20b4e047 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,6 +1,6 @@ # Development Guide -1. [How to set up your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) +1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) 2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (which files to edit, debugging help, etc) 3. [How to make a PR](#3-how-to-make-a-pr) (our expectations for a PR) @@ -16,12 +16,12 @@ All the setup options are for your convenience. If you can get DimOS running on ### Why pick this option? (pros/cons/when-to-use) -* Downside: mutates your global system, which can create side effects and make it less reliable +* Downside: mutates your global system, causing (and receiving) side effects causes it to be unreliable * Upside: Often good for a quick hack or exploring * Upside: Sometimes easier for CUDA/GPU acceleration * Use when: you understand system package management (arch linux user) or you don't care about making changes to your system -### How to set up DimOS +### How to setup DimOS ```bash # System dependencies @@ -228,13 +228,15 @@ This will save the rerun data to `rerun.json` in the current directory. ## Where is `` located? (Architecture) -* If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) -* If you want to add a camera driver see [depth_camera_integration.md](/docs/development/depth_camera_integration.md) -* For edits to manipulation see [manipulation](/dimos/hardware/manipulators/README.md) and the related modules under `dimos/manipulation/`. + +* If you want to add a `dimos run ` command see [dimos_run.md](/dimos/robot/cli/README.md) +* If you want to add a camera driver see [depth_camera_integration.md](/docs/depth_camera_integration.md) + +* For edits to manipulation see [manipulation.md](/dimos/hardware/manipulators/README.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. -* `dimos/msgs/`: If you're trying to find a message type to send over a stream, look here. +* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. * `dimos/dashboard/`: Contains code related to visualization. * `dimos/protocol/`: Defines low level stuff for communication between modules. * See `dimos/` for the remainder @@ -256,7 +258,7 @@ pytest # run all tests at or below the current directory | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be run in CI and for cases that require hardware or visual inspection (pointcloud merging visualization, etc). +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) You can enable a tag by selecting -m - these are configured in `./pyproject.toml` @@ -266,12 +268,12 @@ You can enable a tag by selecting -m - these are configured in `./pyp - Open the PR against the `dev` branch (not `main`). - **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. -- If you're writing documentation, see [writing docs](/docs/development/writing_docs/README.md) +- If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. - If you get mypy errors, please fix them. Don't just add # type: ignore. Please first understand why mypy is complaining and try to fix it. It's only okay to ignore if the issue cannot be fixed. - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large file management](/docs/data.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Did we mention smaller PR's are better? Smaller PR's are better. diff --git a/docs/development/depth_camera_integration.md b/docs/development/depth_camera_integration.md index 3ff162d646..4fca10da4e 100644 --- a/docs/development/depth_camera_integration.md +++ b/docs/development/depth_camera_integration.md @@ -6,7 +6,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## Add a New Depth Camera 1) **Create a new driver module** - - Path: `dimos/hardware/sensors/camera//camera.py` + - Path: `dimos/dimos/hardware/sensors/camera//camera.py` - Export a blueprint in `/__init__.py` (match the `realsense` / `zed` pattern). 2) **Define config** @@ -57,7 +57,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## TF: Required Frames and Transforms -Frame names are defined by the abstract depth camera spec (`dimos/hardware/sensors/camera/spec.py`). +Frame names are defined by the abstract depth camera spec (`dimos/dimos/hardware/sensors/camera/spec.py`). Use the properties below to ensure consistent naming: - `_camera_link`: base link for the camera module (usually `{camera_name}_link`) @@ -111,8 +111,8 @@ For `ObjectSceneRegistrationModule`, the required inputs are: - Overlay annotations and aggregated pointclouds See: -- `dimos/perception/object_scene_registration.py` -- `dimos/perception/demo_object_scene_registration.py` +- `dimos/dimos/perception/object_scene_registration.py` +- `dimos/dimos/perception/demo_object_scene_registration.py` Quick wiring example: @@ -143,5 +143,5 @@ Install Foxglove from: - **Skills** are callable methods (decorated with `@skill`) exposed by `SkillModule` for agents. Reference: -- Modules overview: `/docs/concepts/modules.md` -- TF fundamentals: `/docs/api/transforms.md` +- Modules overview: `dimos/docs/concepts/modules.md` +- TF fundamentals: `dimos/docs/api/transforms.md` diff --git a/docs/development/dimos_run.md b/docs/development/dimos_run.md index 9fb0b5845e..63087f48b8 100644 --- a/docs/development/dimos_run.md +++ b/docs/development/dimos_run.md @@ -1,20 +1,6 @@ -# DimOS Run +# Robot CLI -#### Warning: If you just want to run a blueprint you don't need to add it to `dimos run`: - -`your_code.py` -```py -from dimos.robot.unitree_webrtc.unitree_go2_blueprints import basic as example_blueprint - -if __name__ == "__main__": - example_blueprint.build().loop() -``` - -```sh -python ./your_code.py -``` - -## Usage +To avoid having so many runfiles, I created a common script to run any blueprint. For example, to run the standard Unitree Go2 blueprint run: @@ -34,7 +20,7 @@ You can dynamically connect additional modules. For example: dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill ``` -## Adding your own +## Definitions Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: diff --git a/docs/development/writing_docs/README.md b/docs/development/writing_docs/README.md deleted file mode 100644 index abced8b706..0000000000 --- a/docs/development/writing_docs/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Writing Docs - -Note: as of the DimOS beta, not all existing docs conform to this guide, but newly added docs should. - -## Need-to-know Things - -1. Where to put your docs. - - Some docs are under `docs/` (like this one) but others are stored in the actual codebase, like `dimos/robot/drone/README.md`. - - If your docs have code examples and are somewhere under `docs/`, those code examples must be executable. See [codeblocks guide](/docs/development/writing_docs/codeblocks.md) for details and instructions on how to execute your code examples. - - If your docs nicely *introduce* a new API, or they are a tutorial, then put them in `docs/concepts/` (even if they are about a specific API). - - If the docs are highly technical or exhaustive there are a three options: - - If your docs are about a user-facing API (ex: the reader can follow your instructions without cloning dimos) then put them in `docs/api/`. - - Otherwise (if the reader is modifying their own copy of the dimos codebase) then your docs have two options: - 1. You can choose to store your docs next to relevant python files (ex: `dimos/robot/drone/README.md`), and we are less strict about the contents (code examples don't need to be executable) **BUT**, you need to edit something in `docs/development/` or `docs/api/` to add a reference/link to those docs (don't create "dangling" documentation). - 2. Alternatively, you can put your docs in `docs/development/`. Code examples there should be executable. -2. Even if you know how to link to other docs, read our [how we do doc linking guide](/docs/development/writing_docs/doclinks.md). -3. Even if you know how to create diagrams on your own, read our [how we do diagrams guide](/docs/development/writing_docs/diagram_practices.md). diff --git a/docs/development/writing_docs/assets/pikchr_basic.svg b/docs/development/writing_docs/assets/pikchr_basic.svg deleted file mode 100644 index 9dee0bfdec..0000000000 --- a/docs/development/writing_docs/assets/pikchr_basic.svg +++ /dev/null @@ -1,12 +0,0 @@ - - -Step 1 - - - -Step 2 - - - -Step 3 - diff --git a/docs/development/writing_docs/assets/pikchr_sizing.svg b/docs/development/writing_docs/assets/pikchr_sizing.svg deleted file mode 100644 index 9d17f571d1..0000000000 --- a/docs/development/writing_docs/assets/pikchr_sizing.svg +++ /dev/null @@ -1,13 +0,0 @@ - - -short - - - -.subscribe() - - - -two lines -of text - diff --git a/docs/development/writing_docs/codeblocks.md b/docs/development/writing_docs/codeblocks.md index a928f1ceb4..323f1c0c50 100644 --- a/docs/development/writing_docs/codeblocks.md +++ b/docs/development/writing_docs/codeblocks.md @@ -1,97 +1,16 @@ -# Code Blocks Must Be Executable +# Executable Code Blocks We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. ## Golden Rule -**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. - -## Installation - -
-Click to see full installation instructions - -### Nix (recommended) - -```sh skip -# (assuming you have nix) - -# Run directly from GitHub -nix run github:leshy/md-babel-py -- run README.md --stdout - -# run locally -nix run . -- run README.md --stdout -``` - -### Docker - -```sh skip -# Pull from Docker Hub -docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout - -# Or build locally via Nix -nix build '.#docker' # builds tarball to ./result -docker load < result # loads image from tarball -docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout -``` - -### pipx - -```sh skip -pipx install md-babel-py -# or: uv pip install md-babel-py -md-babel-py run README.md --stdout -``` - -If not using nix or docker, evaluators require system dependencies: - -| Language | System packages | -|-----------|-----------------------------| -| python | python3 | -| node | nodejs | -| dot | graphviz | -| asymptote | asymptote, texlive, dvisvgm | -| pikchr | pikchr | -| openscad | openscad, xvfb, imagemagick | -| diagon | diagon | - -```sh skip -# Arch Linux -sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick - -# Debian/Ubuntu -sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad -``` - -Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. - -## Usage - -```sh skip -# Edit file in-place -md-babel-py run document.md - -# Output to separate file -md-babel-py run document.md --output result.md - -# Print to stdout -md-babel-py run document.md --stdout - -# Only run specific languages -md-babel-py run document.md --lang python,sh - -# Dry run - show what would execute -md-babel-py run document.md --dry-run -``` - -
- +**All code blocks must be executable.** Never write illustrative/pseudo code blocks. If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. ## Running ```sh skip -md-babel-py run document.md # edit in-place -md-babel-py run document.md --stdout # preview to stdout +md-babel-py run document.md # edit in-place +md-babel-py run document.md --stdout # preview to stdout md-babel-py run document.md --dry-run # show what would run ``` @@ -193,3 +112,203 @@ plt.savefig('{output}', transparent=True) ![output](assets/matplotlib-demo.svg) + +### Pikchr + +SQLite's diagram language: + +
+diagram source + +```pikchr fold output=assets/pikchr-demo.svg +color = white +fill = none +linewid = 0.4in + +# Input file +In: file "README.md" fit +arrow + +# Processing +Parse: box "Parse" rad 5px fit +arrow +Exec: box "Execute" rad 5px fit + +# Fan out to languages +arrow from Exec.e right 0.3in then up 0.4in then right 0.3in +Sh: oval "Shell" fit +arrow from Exec.e right 0.3in then right 0.3in +Node: oval "Node" fit +arrow from Exec.e right 0.3in then down 0.4in then right 0.3in +Py: oval "Python" fit + +# Merge back +X: dot at (Py.e.x + 0.3in, Node.e.y) invisible +line from Sh.e right until even with X then down to X +line from Node.e to X +line from Py.e right until even with X then up to X +Out: file "README.md" fit with .w at (X.x + 0.3in, X.y) +arrow from X to Out.w +``` + +
+ + +![output](assets/pikchr-demo.svg) + +### Asymptote + +Vector graphics: + +```asymptote output=assets/histogram.svg +import graph; +import stats; + +size(400,200,IgnoreAspect); +defaultpen(white); + +int n=10000; +real[] a=new real[n]; +for(int i=0; i < n; ++i) a[i]=Gaussrand(); + +draw(graph(Gaussian,min(a),max(a)),orange); + +int N=bins(a); + +histogram(a,min(a),max(a),N,normalize=true,low=0,rgb(0.4,0.6,0.8),rgb(0.2,0.4,0.6),bars=true); + +xaxis("$x$",BottomTop,LeftTicks,p=white); +yaxis("$dP/dx$",LeftRight,RightTicks(trailingzero),p=white); +``` + + +![output](assets/histogram.svg) + +### Graphviz + +```dot output=assets/graph.svg +A -> B -> C +A -> C +``` + + +![output](assets/graph.svg) + +### OpenSCAD + +```openscad output=assets/cube-sphere.png +cube([10, 10, 10]); +sphere(r=7); +``` + + +![output](assets/cube-sphere.png) + +### Diagon + +ASCII art diagrams: + +```diagon mode=Math +1 + 1/2 + sum(i,0,10) +``` + + +``` + 10 + ___ + 1 ╲ +1 + ─ + ╱ i + 2 ‾‾‾ + 0 +``` + +```diagon mode=GraphDAG +A -> B -> C +A -> C +``` + + +``` +┌───┐ +│A │ +└┬─┬┘ + │┌▽┐ + ││B│ + │└┬┘ +┌▽─▽┐ +│C │ +└───┘ +``` + +## Install + +### Nix (recommended) + +```sh skip +# Run directly from GitHub +nix run github:leshy/md-babel-py -- run README.md --stdout + +# Or clone and run locally +nix run . -- run README.md --stdout +``` + +### Docker + +```sh skip +# Pull from Docker Hub +docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout + +# Or build locally via Nix +nix build .#docker # builds tarball to ./result +docker load < result # loads image from tarball +docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout +``` + +### pipx + +```sh skip +pipx install md-babel-py +# or: uv pip install md-babel-py +md-babel-py run README.md --stdout +``` + +If not using nix or docker, evaluators require system dependencies: + +| Language | System packages | +|-----------|-----------------------------| +| python | python3 | +| node | nodejs | +| dot | graphviz | +| asymptote | asymptote, texlive, dvisvgm | +| pikchr | pikchr | +| openscad | openscad, xvfb, imagemagick | +| diagon | diagon | + +```sh skip +# Arch Linux +sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick + +# Debian/Ubuntu +sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad +``` + +Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. + +## Usage + +```sh skip +# Edit file in-place +md-babel-py run document.md + +# Output to separate file +md-babel-py run document.md --output result.md + +# Print to stdout +md-babel-py run document.md --stdout + +# Only run specific languages +md-babel-py run document.md --lang python,sh + +# Dry run - show what would execute +md-babel-py run document.md --dry-run +``` diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index 81f3da73a5..94dd64b72a 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -1,9 +1,59 @@ -We have many diagramming tools. View source code of this page to see examples. -# How to make diagrams +# Code Blocks + +**All code blocks must be executable.** +Never write illustrative/pseudocode blocks. +If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. + +After writing a code block in your markdown file, you can run it by executing: +```bash +md-babel-py run document.md +``` + +More information on this tool is in [codeblocks](/docs/agents/docs/codeblocks.md). + + +# Code or Docs Links + +After adding a link to a doc, run + +```bash +doclinks document.md +``` + +### Code file references +```markdown +See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. +``` + +After running doclinks, it becomes: +```markdown +See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. +``` + +### Symbol auto-linking +Mention a symbol on the same line to auto-link to its line number: +```markdown +The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). +``` + +Becomes: +```markdown +The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). +``` +### Doc-to-doc references +Use `.md` as the link target: +```markdown +See [Configuration](/docs/api/configuration.md) for more details. +``` + +Becomes: +```markdown +See [Configuration](/docs/concepts/configuration.md) for more details. +``` + +More information on this is in [doclinks](/docs/agents/docs/doclinks.md). -1. First define a diagram using a codeblock (examples below). See [Pikchr](https://pikchr.org/) for more details on syntax. -2. Then use the cli tool `md-babel-py` (ex: `md-babel-py run README.md`) to generate the diagram. See [codeblocks.md](/docs/development/writing_docs/codeblocks.md) for how to get the `md-babel-py` cli tool. # Pikchr diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md index e9a6ee9d8c..dce2e67fec 100644 --- a/docs/development/writing_docs/doclinks.md +++ b/docs/development/writing_docs/doclinks.md @@ -1,27 +1,6 @@ -# Use Doclinks to Resolve file references +# doclinks -## Syntax - - -| Pattern | Example | -|-------------|-----------------------------------------------------| -| Code file | `[`service/spec.py`]()` → resolves path | -| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | -| Doc link | `[Configuration](.md)` → resolves to doc | - - -## Usage - -```bash -bin/doclinks docs/guide.md # single file -bin/doclinks docs/ # directory -bin/doclinks --dry-run ... # preview only -``` - -## Full Documentation - -
-Click to see full documentation +A Markdown link resolver that automatically fills in correct file paths for code references in documentation. ## What it does @@ -60,26 +39,26 @@ See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. ```bash # Process a single file -bin/doclinks docs/guide.md +doclinks docs/guide.md # Process a directory recursively -bin/doclinks docs/ +doclinks docs/ # Relative links (from doc location) -bin/doclinks --link-mode relative docs/ +doclinks --link-mode relative docs/ # GitHub links -bin/doclinks --link-mode github \ +doclinks --link-mode github \ --github-url https://github.com/org/repo docs/ # Dry run (preview changes) -bin/doclinks --dry-run docs/ +doclinks --dry-run docs/ # CI check (exit 1 if changes needed) -bin/doclinks --check docs/ +doclinks --check docs/ # Watch mode (auto-update on changes) -bin/doclinks --watch docs/ +doclinks --watch docs/ ``` ## Options @@ -115,5 +94,3 @@ The tool builds an index of all files in the repo. For `/dimos/protocol/service/ - `dimos/protocol/service/spec.py` Use longer paths when multiple files share the same name. - -
From 1c065bbcc252f586f1d5cc55533c96bc8a4976f3 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 20:34:17 -0800 Subject: [PATCH 134/136] Revert "neutralize edits" This reverts commit de3f34da3dc84a8454c2b1a512ba78ccbb616288. --- README.md | 2 +- bin/doclinks | 3 + docs/api/README.md | 11 + docs/api/configuration.md | 2 +- docs/api/sensor_streams/quality_filter.md | 2 +- docs/api/sensor_streams/storage_replay.md | 2 +- docs/api/sensor_streams/temporal_alignment.md | 4 +- docs/api/visualization.md | 2 +- docs/concepts/README.md | 12 + docs/concepts/blueprints.md | 159 +++++++--- docs/concepts/modules.md | 13 +- docs/concepts/transports.md | 14 +- docs/development/README.md | 22 +- docs/development/depth_camera_integration.md | 12 +- docs/development/dimos_run.md | 20 +- docs/development/writing_docs/README.md | 17 ++ .../writing_docs/assets/pikchr_basic.svg | 12 + .../writing_docs/assets/pikchr_sizing.svg | 13 + docs/development/writing_docs/codeblocks.md | 289 ++++++------------ .../writing_docs/diagram_practices.md | 58 +--- docs/development/writing_docs/doclinks.md | 41 ++- 21 files changed, 364 insertions(+), 346 deletions(-) create mode 100755 bin/doclinks create mode 100644 docs/api/README.md create mode 100644 docs/concepts/README.md create mode 100644 docs/development/writing_docs/README.md create mode 100644 docs/development/writing_docs/assets/pikchr_basic.svg create mode 100644 docs/development/writing_docs/assets/pikchr_sizing.svg diff --git a/README.md b/README.md index 5c8ab2128a..ab68e254b8 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Core Features: test dimensional applications in real-time. Control your robot via waypoint, agent query, keyboard, VR, more. - **Modules:** Standalone components (equivalent to ROS nodes) that publish and subscribe to typed - In/Out streams that communicate over DimOS transports. The building blocks of Dimensional. + In/Out streams that communicate over DimOS transports. The primary components of Dimensional. - **Agents (experimental):** DimOS agents understand physical space, subscribe to sensor streams, and call **physical** tools. Emergence appears when agents have physical agency. - **MCP (experimental):** Vibecode robots by giving your AI editor (Cursor, Claude Code) MCP access to run diff --git a/bin/doclinks b/bin/doclinks new file mode 100755 index 0000000000..5dee1c69b0 --- /dev/null +++ b/bin/doclinks @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@" diff --git a/docs/api/README.md b/docs/api/README.md new file mode 100644 index 0000000000..1a14f83640 --- /dev/null +++ b/docs/api/README.md @@ -0,0 +1,11 @@ +# API + +Note: Please see [Concepts](/docs/concepts/README.md) before diving into these API docs. These docs are designed to be technical, cover specific tooling, and give examples of using those tools (argument types) rather than explaining when to use the tool. + + +# Table of Contents + +- [adding configs to your modules](./configuration.md) +- [controlling how visualizations are rendered](./visualization.md) +- [understanding sensor streams](./sensor_streams/README.md) +- [creating coordinate frames and transforms for your hardware](./transforms.md) diff --git a/docs/api/configuration.md b/docs/api/configuration.md index a9c8de0268..162c343ffa 100644 --- a/docs/api/configuration.md +++ b/docs/api/configuration.md @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something' # Configurable Modules -[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids, etc. +[Modules](/docs/concepts/modules.md) inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame IDs, etc. ```python from dataclasses import dataclass diff --git a/docs/api/sensor_streams/quality_filter.md b/docs/api/sensor_streams/quality_filter.md index 26d40733fd..01d3b9367d 100644 --- a/docs/api/sensor_streams/quality_filter.md +++ b/docs/api/sensor_streams/quality_filter.md @@ -50,7 +50,7 @@ Qualities: [0.9] For camera streams, we provide `sharpness_barrier` which uses the image's sharpness score. -Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Replay](/docs/old/testing_stream_reply.md) toolkit, which provides access to recorded robot data: +Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Storage & Replay](/docs/api/sensor_streams/storage_replay.md) toolkit, which provides access to recorded robot data: ```python session=qb from dimos.utils.testing import TimedSensorReplay diff --git a/docs/api/sensor_streams/storage_replay.md b/docs/api/sensor_streams/storage_replay.md index 66e913b197..1a31591736 100644 --- a/docs/api/sensor_streams/storage_replay.md +++ b/docs/api/sensor_streams/storage_replay.md @@ -202,7 +202,7 @@ Each pickle file contains a tuple `(timestamp, data)`: Files are numbered sequentially: `000.pickle`, `001.pickle`, etc. -Recordings are stored in the `data/` directory. See [Data Loading](/docs/data.md) for how data storage works, including Git LFS handling for large datasets. +Recordings are stored in the `data/` directory. See [Data Loading](/docs/development/large_file_management.md) for how data storage works, including Git LFS handling for large datasets. ## API Reference diff --git a/docs/api/sensor_streams/temporal_alignment.md b/docs/api/sensor_streams/temporal_alignment.md index b552ac54cc..66230c9d54 100644 --- a/docs/api/sensor_streams/temporal_alignment.md +++ b/docs/api/sensor_streams/temporal_alignment.md @@ -34,7 +34,7 @@ Below we set up replay of real camera and lidar data from the Unitree Go2 robot.
Stream Setup -You can read more about [sensor storage here](storage_replay.md) and [LFS data store here](/docs/data.md). +You can read more about [sensor storage here](storage_replay.md) and [LFS data storage here](/docs/development/large_file_management.md). ```python session=align no-result from reactivex import Subject @@ -70,7 +70,7 @@ lidar_stream = lidar_replay.stream(from_timestamp=seek_ts, duration=2.0).pipe(
-Streams would normally come from an actual robot into your module via `IN` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. +Streams would normally come from an actual robot into your module via `In` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this. Assume we have them. Let's align them. diff --git a/docs/api/visualization.md b/docs/api/visualization.md index 51fa20d655..08919b7a73 100644 --- a/docs/api/visualization.md +++ b/docs/api/visualization.md @@ -54,7 +54,7 @@ VIEWER_BACKEND=foxglove dimos run unitree-go2 - Foxglove bridge on ws://localhost:8765 - No Rerun (saves resources) - Better performance with larger maps/higher resolution -- Open layout: `dimos/assets/foxglove_dashboards/go2.json` +- Open layout: `assets/foxglove_dashboards/old/foxglove_unitree_lcm_dashboard.json` --- diff --git a/docs/concepts/README.md b/docs/concepts/README.md new file mode 100644 index 0000000000..0f82783099 --- /dev/null +++ b/docs/concepts/README.md @@ -0,0 +1,12 @@ +# Concepts + +This page explains general concepts. For specific API docs see [the API reference](/docs/api/README.md). + +## Table of Contents + +- [Modules](/docs/concepts/modules.md): The primary units of deployment in DimOS, modules run in parallel and are python classes. +- [Streams](/docs/api/sensor_streams/README.md): How modules communicate, a Pub / Sub system. +- [Blueprints](/docs/concepts/blueprints.md): a way to group modules together and define their connections to each other. +- [RPC](/docs/concepts/blueprints.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data). +- [Skills](/docs/concepts/blueprints.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI). +- Agents: AI that has an objective, access to stream data, and is capable of calling skills as tools. diff --git a/docs/concepts/blueprints.md b/docs/concepts/blueprints.md index 0a3e2ceaf5..686aff60d5 100644 --- a/docs/concepts/blueprints.md +++ b/docs/concepts/blueprints.md @@ -6,19 +6,26 @@ You don't typically want to run a single module, so multiple blueprints are hand You create a `ModuleBlueprintSet` from a single module (say `ConnectionModule`) with: -```python +```python session=blueprint-ex1 +from dimos.core.blueprints import create_module_blueprint +from dimos.core import Module, rpc + +class ConnectionModule(Module): + def __init__(self, arg1, arg2, kwarg='value') -> None: + super().__init__() + blueprint = create_module_blueprint(ConnectionModule, 'arg1', 'arg2', kwarg='value') ``` -But the same thing can be acomplished more succinctly as: +But the same thing can be accomplished more succinctly as: -```python +```python session=blueprint-ex1 connection = ConnectionModule.blueprint ``` Now you can create the blueprint with: -```python +```python session=blueprint-ex1 blueprint = connection('arg1', 'arg2', kwarg='value') ``` @@ -26,7 +33,23 @@ blueprint = connection('arg1', 'arg2', kwarg='value') You can link multiple blueprints together with `autoconnect`: -```python +```python session=blueprint-ex1 +from dimos.core.blueprints import autoconnect + +class Module1(Module): + def __init__(self, arg1) -> None: + super().__init__() + +class Module2(Module): + ... + +class Module3(Module): + ... + +module1 = Module1.blueprint +module2 = Module2.blueprint +module3 = Module3.blueprint + blueprint = autoconnect( module1(), module2(), @@ -36,7 +59,16 @@ blueprint = autoconnect( `blueprint` itself is a `ModuleBlueprintSet` so you can link it with other modules: -```python +```python session=blueprint-ex1 +class Module4(Module): + ... + +class Module5(Module): + ... + +module4 = Module4.blueprint +module5 = Module5.blueprint + expanded_blueprint = autoconnect( blueprint, module4(), @@ -50,11 +82,11 @@ Blueprints are frozen data classes, and `autoconnect()` always constructs an exp If the same module appears multiple times in `autoconnect`, the **later blueprint wins** and overrides earlier ones: -```python +```python session=blueprint-ex1 blueprint = autoconnect( - module_a(arg1=1), - module_b(), - module_a(arg1=2), # This one is used, the first is discarded + module1(arg1=1), + module2(), + module1(arg1=2), # This one is used, the first is discarded ) ``` @@ -64,14 +96,20 @@ This is so you can "inherit" from one blueprint but override something you need Imagine you have this code: -```python +```python session=blueprint-ex1 +from functools import partial + +from dimos.core.blueprints import create_module_blueprint, autoconnect +from dimos.core import Module, rpc, Out, In +from dimos.msgs.sensor_msgs import Image + class ModuleA(Module): image: Out[Image] - start_explore: Out[Bool] + start_explore: Out[bool] class ModuleB(Module): image: In[Image] - begin_explore: In[Bool] + begin_explore: In[bool] module_a = partial(create_module_blueprint, ModuleA) module_b = partial(create_module_blueprint, ModuleB) @@ -95,24 +133,37 @@ By default `LCMTransport` is used if the object supports `lcm_encode`. If it doe You can override transports with the `transports` method. It returns a new blueprint in which the override is set. -```python -blueprint = autoconnect(...) -expanded_blueprint = autoconnect(blueprint, ...) -blueprint = blueprint.transports({ +```python session=blueprint-ex1 +from dimos.core.transport import pSHMTransport, pLCMTransport + +base_blueprint = autoconnect( + module1(arg1=1), + module2(), +) +expanded_blueprint = autoconnect( + base_blueprint, + module4(), + module5(), +) +base_blueprint = base_blueprint.transports({ ("image", Image): pSHMTransport( - "/go2/color_image", default_capacity=DEFAULT_CAPACITY_COLOR_IMAGE + "/go2/color_image", default_capacity=1920 * 1080 * 3, # 1920x1080 frame x 3 (RGB) x uint8 ), - ("start_explore", Bool): pLCMTransport(), + ("start_explore", bool): pLCMTransport("/start_explore"), }) ``` -Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `blueprint`, not the second. +Note: `expanded_blueprint` does not get the transport overrides because it's created from the initial value of `base_blueprint`, not the second. ## Remapping connections Sometimes you need to rename a connection to match what other modules expect. You can use `remappings` to rename module connections: -```python +```python session=blueprint-ex2 +from dimos.core.blueprints import autoconnect +from dimos.core import Module, rpc, Out, In +from dimos.msgs.sensor_msgs import Image + class ConnectionModule(Module): color_image: Out[Image] # Outputs on 'color_image' @@ -139,12 +190,11 @@ After remapping: If you want to override the topic, you still have to do it manually: -```python -blueprint -.remappings([ +```python session=blueprint-ex2 +from dimos.core.transport import LCMTransport +blueprint.remappings([ (ConnectionModule, 'color_image', 'rgb_image'), -]) -.transports({ +]).transports({ ("rgb_image", Image): LCMTransport("/custom/rgb/image", Image), }) ``` @@ -153,7 +203,10 @@ blueprint Each module can optionally take a `global_config` option in `__init__`. E.g.: -```python +```python session=blueprint-ex3 +from dimos.core import Module, rpc +from dimos.core.global_config import GlobalConfig + class ModuleA(Module): def __init__(self, global_config: GlobalConfig | None = None): @@ -162,15 +215,17 @@ class ModuleA(Module): The config is normally taken from .env or from environment variables. But you can specifically override the values for a specific blueprint: -```python -blueprint = blueprint.global_config(n_dask_workers=8) +```python session=blueprint-ex3 +blueprint = ModuleA.blueprint().global_config(n_dask_workers=8) ``` ## Calling the methods of other modules Imagine you have this code: -```python +```python session=blueprint-ex3 +from dimos.core import Module, rpc + class ModuleA(Module): @rpc @@ -186,7 +241,9 @@ And you want to call `ModuleA.get_time` in `ModuleB.request_the_time`. To do this, you can request a link to the method you want to call in `rpc_calls`. Calling `get_time_rcp` will call the original `ModuleA.get_time`. -```python +```python session=blueprint-ex3 +from dimos.core import Module, rpc + class ModuleB(Module): rpc_calls: list[str] = [ "ModuleA.get_time", @@ -199,8 +256,10 @@ class ModuleB(Module): You can also request multiple methods at a time: -```python -method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") +```python session=blueprint-ex3 +class ModuleB(Module): + def request_the_time(self) -> None: + method1_rpc, method2_rpc = self.get_rpc_calls("ModuleX.m1", "ModuleX.m2") ``` ## Alternative RPC calls @@ -209,7 +268,10 @@ There is an alternative way of receiving RPC methods. It is useful when you want You can use it by defining a method like `set__`: -```python +```python session=blueprint-ex3 +from dimos.core import Module, rpc +from dimos.core.rpc_client import RpcCall + class ModuleB(Module): @rpc # Note that it has to be an rpc method. def set_ModuleA_get_time(self, rpc_call: RpcCall) -> None: @@ -228,12 +290,16 @@ In the previous examples, you can only call methods in a module called `ModuleA` You can do so by extracting the common interface as an `ABC` (abstract base class) and linking to the `ABC` instead one particular class. -```python +```python session=blueprint-ex3 +from abc import ABC, abstractmethod +from dimos.core.blueprints import autoconnect +from dimos.core import Module, rpc + class TimeInterface(ABC): @abstractmethod def get_time(self): ... -class ProperTime(TimeInterface): +class ProperTime(Module, TimeInterface): def get_time(self): return "13:00" @@ -254,7 +320,7 @@ class ModuleB(Module): The actual method that you get in `get_time_rpc` depends on which module is deployed. If you deploy `ProperTime`, you get `ProperTime.get_time`: -```python +```python session=blueprint-ex3 blueprint = autoconnect( ProperTime.blueprint(), # get_rpc_calls("TimeInterface.get_time") returns ProperTime.get_time @@ -268,7 +334,13 @@ If both are deployed, the blueprint will throw an error because it's ambiguous. Skills have to be registered with `AgentSpec.register_skills(self)`. -```python +```python session=blueprint-ex4 +from dimos.core import Module, rpc +from dimos.core.skill_module import SkillModule +from dimos.protocol.skill.skill import skill +from dimos.core.rpc_client import RpcCall +from dimos.core.global_config import GlobalConfig + class SomeSkill(Module): @skill @@ -290,7 +362,10 @@ class SomeSkill(Module): Or, you can avoid all of this by inheriting from `SkillModule` which does the above automatically: -```python +```python session=blueprint-ex4 +from dimos.core.skill_module import SkillModule +from dimos.protocol.skill.skill import skill + class SomeSkill(SkillModule): @skill @@ -302,8 +377,8 @@ class SomeSkill(SkillModule): All you have to do to build a blueprint is call: -```python -module_coordinator = blueprint.build(global_config=config) +```python session=blueprint-ex4 +module_coordinator = SomeSkill.blueprint().build(global_config=GlobalConfig()) ``` This returns a `ModuleCoordinator` instance that manages all deployed modules. @@ -312,7 +387,7 @@ This returns a `ModuleCoordinator` instance that manages all deployed modules. You can block the thread until it exits with: -```python +```python session=blueprint-ex4 module_coordinator.loop() ``` diff --git a/docs/concepts/modules.md b/docs/concepts/modules.md index ee7fbaf2c9..344efa0774 100644 --- a/docs/concepts/modules.md +++ b/docs/concepts/modules.md @@ -1,5 +1,5 @@ -# Dimos Modules +# DimOS Modules Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages. @@ -47,8 +47,8 @@ print(CameraModule.io()) ├─ color_image: Image ├─ camera_info: CameraInfo │ - ├─ RPC set_transport(stream_name: str, transport: Transport) -> bool ├─ RPC start() + ├─ RPC stop() │ ├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image) ``` @@ -58,9 +58,9 @@ We can see that the camera module outputs two streams: - `color_image` with [sensor_msgs.Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) type - `camera_info` with [sensor_msgs.CameraInfo](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) type -It offers two RPC calls: `start()` and `stop()`. +It offers two RPC calls: `start()` and `stop()` (lifecycle methods). -As well as an agentic [Skill](skills.md) called `video_stream` (more about this later, in [Skills Tutorial](skills.md)). +It also exposes an agentic [skill](/docs/concepts/blueprints.md#defining-skills) called `video_stream` (more on skills in the Blueprints guide). We can start this module and explore the output of its streams in real time (this will use your webcam). @@ -120,7 +120,7 @@ print(Detection2DModule.io()) ├─ RPC stop() -> None ``` -TODO: add easy way to print config + Looks like the detector just needs an image input and outputs some sort of detection and annotation messages. Let's connect it to a camera. @@ -174,3 +174,6 @@ to_svg(agentic, "assets/go2_agentic.svg") ![output](assets/go2_agentic.svg) + + +To see more information on how to use Blueprints, see [Blueprints](/docs/concepts/blueprints.md). diff --git a/docs/concepts/transports.md b/docs/concepts/transports.md index 62279b6baf..6c9e5fca99 100644 --- a/docs/concepts/transports.md +++ b/docs/concepts/transports.md @@ -18,11 +18,17 @@ print(inspect.getsource(PubSub.publish)) print(inspect.getsource(PubSub.subscribe)) ``` - ``` -Session process exited unexpectedly: -/home/lesh/coding/dimos/.venv/bin/python3: No module named md_babel_py.session_server - +@abstractmethod +def publish(self, topic: TopicT, message: MsgT) -> None: + """Publish a message to a topic.""" + ... +@abstractmethod +def subscribe( + self, topic: TopicT, callback: Callable[[MsgT, TopicT], None] +) -> Callable[[], None]: + """Subscribe to a topic with a callback. returns unsubscribe function""" + ... ``` Key points: diff --git a/docs/development/README.md b/docs/development/README.md index 9517fda6a1..453ec5dea0 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,6 +1,6 @@ # Development Guide -1. [How to setup your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) +1. [How to set up your system](#1-setup) (pick one: system install, nix flake + direnv, pure nix flake) 2. [How to hack on DimOS](#2-how-to-hack-on-dimos) (which files to edit, debugging help, etc) 3. [How to make a PR](#3-how-to-make-a-pr) (our expectations for a PR) @@ -16,12 +16,12 @@ All the setup options are for your convenience. If you can get DimOS running on ### Why pick this option? (pros/cons/when-to-use) -* Downside: mutates your global system, causing (and receiving) side effects causes it to be unreliable +* Downside: mutates your global system, which can create side effects and make it less reliable * Upside: Often good for a quick hack or exploring * Upside: Sometimes easier for CUDA/GPU acceleration * Use when: you understand system package management (arch linux user) or you don't care about making changes to your system -### How to setup DimOS +### How to set up DimOS ```bash # System dependencies @@ -228,15 +228,13 @@ This will save the rerun data to `rerun.json` in the current directory. ## Where is `` located? (Architecture) - -* If you want to add a `dimos run ` command see [dimos_run.md](/dimos/robot/cli/README.md) -* If you want to add a camera driver see [depth_camera_integration.md](/docs/depth_camera_integration.md) - -* For edits to manipulation see [manipulation.md](/dimos/hardware/manipulators/README.md) and [manipulation base](/dimos/hardware/manipulators/base/component_based_architecture.md) +* If you want to add a `dimos run ` command see [dimos_run.md](/docs/development/dimos_run.md) +* If you want to add a camera driver see [depth_camera_integration.md](/docs/development/depth_camera_integration.md) +* For edits to manipulation see [manipulation](/dimos/hardware/manipulators/README.md) and the related modules under `dimos/manipulation/`. * `dimos/core/`: Is where stuff like `Module`, `In`, `Out`, and `RPC` live. * `dimos/robot/`: Robot-specific modules live here. * `dimos/hardware/`: Are for sensors, end-effectors, and related individual hardware pieces. -* `dimos/msgs/`: If you're trying to find a type to send a type over a stream, look here. +* `dimos/msgs/`: If you're trying to find a message type to send over a stream, look here. * `dimos/dashboard/`: Contains code related to visualization. * `dimos/protocol/`: Defines low level stuff for communication between modules. * See `dimos/` for the remainder @@ -258,7 +256,7 @@ pytest # run all tests at or below the current directory | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be ran in CI and when casually developing, something that requires hardware or visual inspection (pointcloud merging vis etc) +We use tags for special tests, like `vis` or `tool` for things that aren't meant to be run in CI and for cases that require hardware or visual inspection (pointcloud merging visualization, etc). You can enable a tag by selecting -m - these are configured in `./pyproject.toml` @@ -268,12 +266,12 @@ You can enable a tag by selecting -m - these are configured in `./pyp - Open the PR against the `dev` branch (not `main`). - **No matter what, provide a few-lines that, when run, let a reviewer test the feature you added** (assuming you changed functional python code). - Less changed files = better. -- If you're writing documentation, see [writing docs](/docs/agents/docs/index.md) for how to write code blocks. +- If you're writing documentation, see [writing docs](/docs/development/writing_docs/README.md) - If you get mypy errors, please fix them. Don't just add # type: ignore. Please first understand why mypy is complaining and try to fix it. It's only okay to ignore if the issue cannot be fixed. - If you made a change that is likely going to involve a debate, open the github UI and add a graphical comment on that code. Justify your choice and explain downsides of alternatives. - We don't require 100% test coverage, but if you're making a PR of notable python changes you should probably either have unit tests or good reason why not (ex: visualization stuff is hard to test so we don't). - Have the name of your PR start with `WIP:` if its not ready to merge but you want to show someone the changes. -- If you have large (>500kb) files, see [large file management](/docs/data.md) for how to store and load them (don't just commit them). +- If you have large (>500kb) files, see [large file management](/docs/development/large_file_management.md) for how to store and load them (don't just commit them). - So long as you don't disable pre-commit hooks the formatting, license headers, EOLs, LFS checks, etc will be handled automatically by [pre-commit](https://pre-commit.com). If something goes wrong with the hooks you can run the step manually with `pre-commit run --all-files`. - If you're a new hire at DimOS: - Did we mention smaller PR's are better? Smaller PR's are better. diff --git a/docs/development/depth_camera_integration.md b/docs/development/depth_camera_integration.md index 4fca10da4e..3ff162d646 100644 --- a/docs/development/depth_camera_integration.md +++ b/docs/development/depth_camera_integration.md @@ -6,7 +6,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## Add a New Depth Camera 1) **Create a new driver module** - - Path: `dimos/dimos/hardware/sensors/camera//camera.py` + - Path: `dimos/hardware/sensors/camera//camera.py` - Export a blueprint in `/__init__.py` (match the `realsense` / `zed` pattern). 2) **Define config** @@ -57,7 +57,7 @@ Use this guide to add a new depth camera, wire TF correctly, and publish the req ## TF: Required Frames and Transforms -Frame names are defined by the abstract depth camera spec (`dimos/dimos/hardware/sensors/camera/spec.py`). +Frame names are defined by the abstract depth camera spec (`dimos/hardware/sensors/camera/spec.py`). Use the properties below to ensure consistent naming: - `_camera_link`: base link for the camera module (usually `{camera_name}_link`) @@ -111,8 +111,8 @@ For `ObjectSceneRegistrationModule`, the required inputs are: - Overlay annotations and aggregated pointclouds See: -- `dimos/dimos/perception/object_scene_registration.py` -- `dimos/dimos/perception/demo_object_scene_registration.py` +- `dimos/perception/object_scene_registration.py` +- `dimos/perception/demo_object_scene_registration.py` Quick wiring example: @@ -143,5 +143,5 @@ Install Foxglove from: - **Skills** are callable methods (decorated with `@skill`) exposed by `SkillModule` for agents. Reference: -- Modules overview: `dimos/docs/concepts/modules.md` -- TF fundamentals: `dimos/docs/api/transforms.md` +- Modules overview: `/docs/concepts/modules.md` +- TF fundamentals: `/docs/api/transforms.md` diff --git a/docs/development/dimos_run.md b/docs/development/dimos_run.md index 63087f48b8..9fb0b5845e 100644 --- a/docs/development/dimos_run.md +++ b/docs/development/dimos_run.md @@ -1,6 +1,20 @@ -# Robot CLI +# DimOS Run -To avoid having so many runfiles, I created a common script to run any blueprint. +#### Warning: If you just want to run a blueprint you don't need to add it to `dimos run`: + +`your_code.py` +```py +from dimos.robot.unitree_webrtc.unitree_go2_blueprints import basic as example_blueprint + +if __name__ == "__main__": + example_blueprint.build().loop() +``` + +```sh +python ./your_code.py +``` + +## Usage For example, to run the standard Unitree Go2 blueprint run: @@ -20,7 +34,7 @@ You can dynamically connect additional modules. For example: dimos run unitree-go2 --extra-module llm_agent --extra-module human_input --extra-module navigation_skill ``` -## Definitions +## Adding your own Blueprints can be defined anywhere, but they're all linked together in `dimos/robot/all_blueprints.py`. E.g.: diff --git a/docs/development/writing_docs/README.md b/docs/development/writing_docs/README.md new file mode 100644 index 0000000000..abced8b706 --- /dev/null +++ b/docs/development/writing_docs/README.md @@ -0,0 +1,17 @@ +# Writing Docs + +Note: as of the DimOS beta, not all existing docs conform to this guide, but newly added docs should. + +## Need-to-know Things + +1. Where to put your docs. + - Some docs are under `docs/` (like this one) but others are stored in the actual codebase, like `dimos/robot/drone/README.md`. + - If your docs have code examples and are somewhere under `docs/`, those code examples must be executable. See [codeblocks guide](/docs/development/writing_docs/codeblocks.md) for details and instructions on how to execute your code examples. + - If your docs nicely *introduce* a new API, or they are a tutorial, then put them in `docs/concepts/` (even if they are about a specific API). + - If the docs are highly technical or exhaustive there are a three options: + - If your docs are about a user-facing API (ex: the reader can follow your instructions without cloning dimos) then put them in `docs/api/`. + - Otherwise (if the reader is modifying their own copy of the dimos codebase) then your docs have two options: + 1. You can choose to store your docs next to relevant python files (ex: `dimos/robot/drone/README.md`), and we are less strict about the contents (code examples don't need to be executable) **BUT**, you need to edit something in `docs/development/` or `docs/api/` to add a reference/link to those docs (don't create "dangling" documentation). + 2. Alternatively, you can put your docs in `docs/development/`. Code examples there should be executable. +2. Even if you know how to link to other docs, read our [how we do doc linking guide](/docs/development/writing_docs/doclinks.md). +3. Even if you know how to create diagrams on your own, read our [how we do diagrams guide](/docs/development/writing_docs/diagram_practices.md). diff --git a/docs/development/writing_docs/assets/pikchr_basic.svg b/docs/development/writing_docs/assets/pikchr_basic.svg new file mode 100644 index 0000000000..9dee0bfdec --- /dev/null +++ b/docs/development/writing_docs/assets/pikchr_basic.svg @@ -0,0 +1,12 @@ + + +Step 1 + + + +Step 2 + + + +Step 3 + diff --git a/docs/development/writing_docs/assets/pikchr_sizing.svg b/docs/development/writing_docs/assets/pikchr_sizing.svg new file mode 100644 index 0000000000..9d17f571d1 --- /dev/null +++ b/docs/development/writing_docs/assets/pikchr_sizing.svg @@ -0,0 +1,13 @@ + + +short + + + +.subscribe() + + + +two lines +of text + diff --git a/docs/development/writing_docs/codeblocks.md b/docs/development/writing_docs/codeblocks.md index 323f1c0c50..a928f1ceb4 100644 --- a/docs/development/writing_docs/codeblocks.md +++ b/docs/development/writing_docs/codeblocks.md @@ -1,16 +1,97 @@ -# Executable Code Blocks +# Code Blocks Must Be Executable We use [md-babel-py](https://github.com/leshy/md-babel-py/) to execute code blocks in markdown and insert results. ## Golden Rule -**All code blocks must be executable.** Never write illustrative/pseudo code blocks. If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. +**Never write illustrative/pseudo code blocks.** If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. + +## Installation + +
+Click to see full installation instructions + +### Nix (recommended) + +```sh skip +# (assuming you have nix) + +# Run directly from GitHub +nix run github:leshy/md-babel-py -- run README.md --stdout + +# run locally +nix run . -- run README.md --stdout +``` + +### Docker + +```sh skip +# Pull from Docker Hub +docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout + +# Or build locally via Nix +nix build '.#docker' # builds tarball to ./result +docker load < result # loads image from tarball +docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout +``` + +### pipx + +```sh skip +pipx install md-babel-py +# or: uv pip install md-babel-py +md-babel-py run README.md --stdout +``` + +If not using nix or docker, evaluators require system dependencies: + +| Language | System packages | +|-----------|-----------------------------| +| python | python3 | +| node | nodejs | +| dot | graphviz | +| asymptote | asymptote, texlive, dvisvgm | +| pikchr | pikchr | +| openscad | openscad, xvfb, imagemagick | +| diagon | diagon | + +```sh skip +# Arch Linux +sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick + +# Debian/Ubuntu +sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad +``` + +Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. + +## Usage + +```sh skip +# Edit file in-place +md-babel-py run document.md + +# Output to separate file +md-babel-py run document.md --output result.md + +# Print to stdout +md-babel-py run document.md --stdout + +# Only run specific languages +md-babel-py run document.md --lang python,sh + +# Dry run - show what would execute +md-babel-py run document.md --dry-run +``` + +
+ ## Running ```sh skip -md-babel-py run document.md # edit in-place -md-babel-py run document.md --stdout # preview to stdout +md-babel-py run document.md # edit in-place +md-babel-py run document.md --stdout # preview to stdout md-babel-py run document.md --dry-run # show what would run ``` @@ -112,203 +193,3 @@ plt.savefig('{output}', transparent=True) ![output](assets/matplotlib-demo.svg) - -### Pikchr - -SQLite's diagram language: - -
-diagram source - -```pikchr fold output=assets/pikchr-demo.svg -color = white -fill = none -linewid = 0.4in - -# Input file -In: file "README.md" fit -arrow - -# Processing -Parse: box "Parse" rad 5px fit -arrow -Exec: box "Execute" rad 5px fit - -# Fan out to languages -arrow from Exec.e right 0.3in then up 0.4in then right 0.3in -Sh: oval "Shell" fit -arrow from Exec.e right 0.3in then right 0.3in -Node: oval "Node" fit -arrow from Exec.e right 0.3in then down 0.4in then right 0.3in -Py: oval "Python" fit - -# Merge back -X: dot at (Py.e.x + 0.3in, Node.e.y) invisible -line from Sh.e right until even with X then down to X -line from Node.e to X -line from Py.e right until even with X then up to X -Out: file "README.md" fit with .w at (X.x + 0.3in, X.y) -arrow from X to Out.w -``` - -
- - -![output](assets/pikchr-demo.svg) - -### Asymptote - -Vector graphics: - -```asymptote output=assets/histogram.svg -import graph; -import stats; - -size(400,200,IgnoreAspect); -defaultpen(white); - -int n=10000; -real[] a=new real[n]; -for(int i=0; i < n; ++i) a[i]=Gaussrand(); - -draw(graph(Gaussian,min(a),max(a)),orange); - -int N=bins(a); - -histogram(a,min(a),max(a),N,normalize=true,low=0,rgb(0.4,0.6,0.8),rgb(0.2,0.4,0.6),bars=true); - -xaxis("$x$",BottomTop,LeftTicks,p=white); -yaxis("$dP/dx$",LeftRight,RightTicks(trailingzero),p=white); -``` - - -![output](assets/histogram.svg) - -### Graphviz - -```dot output=assets/graph.svg -A -> B -> C -A -> C -``` - - -![output](assets/graph.svg) - -### OpenSCAD - -```openscad output=assets/cube-sphere.png -cube([10, 10, 10]); -sphere(r=7); -``` - - -![output](assets/cube-sphere.png) - -### Diagon - -ASCII art diagrams: - -```diagon mode=Math -1 + 1/2 + sum(i,0,10) -``` - - -``` - 10 - ___ - 1 ╲ -1 + ─ + ╱ i - 2 ‾‾‾ - 0 -``` - -```diagon mode=GraphDAG -A -> B -> C -A -> C -``` - - -``` -┌───┐ -│A │ -└┬─┬┘ - │┌▽┐ - ││B│ - │└┬┘ -┌▽─▽┐ -│C │ -└───┘ -``` - -## Install - -### Nix (recommended) - -```sh skip -# Run directly from GitHub -nix run github:leshy/md-babel-py -- run README.md --stdout - -# Or clone and run locally -nix run . -- run README.md --stdout -``` - -### Docker - -```sh skip -# Pull from Docker Hub -docker run -v $(pwd):/work lesh/md-babel-py:main run /work/README.md --stdout - -# Or build locally via Nix -nix build .#docker # builds tarball to ./result -docker load < result # loads image from tarball -docker run -v $(pwd):/work md-babel-py:latest run /work/file.md --stdout -``` - -### pipx - -```sh skip -pipx install md-babel-py -# or: uv pip install md-babel-py -md-babel-py run README.md --stdout -``` - -If not using nix or docker, evaluators require system dependencies: - -| Language | System packages | -|-----------|-----------------------------| -| python | python3 | -| node | nodejs | -| dot | graphviz | -| asymptote | asymptote, texlive, dvisvgm | -| pikchr | pikchr | -| openscad | openscad, xvfb, imagemagick | -| diagon | diagon | - -```sh skip -# Arch Linux -sudo pacman -S python nodejs graphviz asymptote texlive-basic openscad xorg-server-xvfb imagemagick - -# Debian/Ubuntu -sudo apt-get install python3 nodejs graphviz asymptote texlive xvfb imagemagick openscad -``` - -Note: pikchr and diagon may need to be built from source. Use Docker or Nix for full evaluator support. - -## Usage - -```sh skip -# Edit file in-place -md-babel-py run document.md - -# Output to separate file -md-babel-py run document.md --output result.md - -# Print to stdout -md-babel-py run document.md --stdout - -# Only run specific languages -md-babel-py run document.md --lang python,sh - -# Dry run - show what would execute -md-babel-py run document.md --dry-run -``` diff --git a/docs/development/writing_docs/diagram_practices.md b/docs/development/writing_docs/diagram_practices.md index 94dd64b72a..81f3da73a5 100644 --- a/docs/development/writing_docs/diagram_practices.md +++ b/docs/development/writing_docs/diagram_practices.md @@ -1,59 +1,9 @@ +We have many diagramming tools. View source code of this page to see examples. -# Code Blocks - -**All code blocks must be executable.** -Never write illustrative/pseudocode blocks. -If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves. - -After writing a code block in your markdown file, you can run it by executing: -```bash -md-babel-py run document.md -``` - -More information on this tool is in [codeblocks](/docs/agents/docs/codeblocks.md). - - -# Code or Docs Links - -After adding a link to a doc, run - -```bash -doclinks document.md -``` - -### Code file references -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -After running doclinks, it becomes: -```markdown -See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. -``` - -### Symbol auto-linking -Mention a symbol on the same line to auto-link to its line number: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` - -Becomes: -```markdown -The `Configurable` class is defined in [`service/spec.py`](/dimos/protocol/service/spec.py#L22). -``` -### Doc-to-doc references -Use `.md` as the link target: -```markdown -See [Configuration](/docs/api/configuration.md) for more details. -``` - -Becomes: -```markdown -See [Configuration](/docs/concepts/configuration.md) for more details. -``` - -More information on this is in [doclinks](/docs/agents/docs/doclinks.md). +# How to make diagrams +1. First define a diagram using a codeblock (examples below). See [Pikchr](https://pikchr.org/) for more details on syntax. +2. Then use the cli tool `md-babel-py` (ex: `md-babel-py run README.md`) to generate the diagram. See [codeblocks.md](/docs/development/writing_docs/codeblocks.md) for how to get the `md-babel-py` cli tool. # Pikchr diff --git a/docs/development/writing_docs/doclinks.md b/docs/development/writing_docs/doclinks.md index dce2e67fec..e9a6ee9d8c 100644 --- a/docs/development/writing_docs/doclinks.md +++ b/docs/development/writing_docs/doclinks.md @@ -1,6 +1,27 @@ -# doclinks +# Use Doclinks to Resolve file references -A Markdown link resolver that automatically fills in correct file paths for code references in documentation. +## Syntax + + +| Pattern | Example | +|-------------|-----------------------------------------------------| +| Code file | `[`service/spec.py`]()` → resolves path | +| With symbol | `Configurable` in `[`spec.py`]()` → adds `#L` | +| Doc link | `[Configuration](.md)` → resolves to doc | + + +## Usage + +```bash +bin/doclinks docs/guide.md # single file +bin/doclinks docs/ # directory +bin/doclinks --dry-run ... # preview only +``` + +## Full Documentation + +
+Click to see full documentation ## What it does @@ -39,26 +60,26 @@ See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation. ```bash # Process a single file -doclinks docs/guide.md +bin/doclinks docs/guide.md # Process a directory recursively -doclinks docs/ +bin/doclinks docs/ # Relative links (from doc location) -doclinks --link-mode relative docs/ +bin/doclinks --link-mode relative docs/ # GitHub links -doclinks --link-mode github \ +bin/doclinks --link-mode github \ --github-url https://github.com/org/repo docs/ # Dry run (preview changes) -doclinks --dry-run docs/ +bin/doclinks --dry-run docs/ # CI check (exit 1 if changes needed) -doclinks --check docs/ +bin/doclinks --check docs/ # Watch mode (auto-update on changes) -doclinks --watch docs/ +bin/doclinks --watch docs/ ``` ## Options @@ -94,3 +115,5 @@ The tool builds an index of all files in the repo. For `/dimos/protocol/service/ - `dimos/protocol/service/spec.py` Use longer paths when multiple files share the same name. + +
From a1ebfab13e52569cf42cfb9029b6676302d73e0c Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 20:42:15 -0800 Subject: [PATCH 135/136] remove unused name --- docs/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/README.md b/docs/development/README.md index 453ec5dea0..03444ae732 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -256,7 +256,7 @@ pytest # run all tests at or below the current directory | Enable stdout in tests | `pytest -s` | | Run tagged tests | `pytest -m ` | -We use tags for special tests, like `vis` or `tool` for things that aren't meant to be run in CI and for cases that require hardware or visual inspection (pointcloud merging visualization, etc). +We use tags for special tests, like `tool` for things that aren't meant to be run in CI and for cases that require hardware or visual inspection (pointcloud merging visualization, etc). You can enable a tag by selecting -m - these are configured in `./pyproject.toml` From f9c99ac470222b09909a99f2680c522bc68ee9b2 Mon Sep 17 00:00:00 2001 From: Jeff Hykin Date: Fri, 23 Jan 2026 22:13:44 -0800 Subject: [PATCH 136/136] test against blueprints.md instead of development.md --- dimos/utils/docs/test_doclinks.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dimos/utils/docs/test_doclinks.py b/dimos/utils/docs/test_doclinks.py index 7313ec3676..016cc30113 100644 --- a/dimos/utils/docs/test_doclinks.py +++ b/dimos/utils/docs/test_doclinks.py @@ -277,7 +277,7 @@ def test_indexes_by_stem(self, doc_index): """Should index docs by lowercase stem.""" assert "configuration" in doc_index assert "modules" in doc_index - assert "development" in doc_index + assert "blueprints" in doc_index def test_case_insensitive(self, doc_index): """Should use lowercase keys.""" @@ -349,8 +349,8 @@ def test_doc_link_github_mode(self, file_index, doc_index): def test_doc_link_relative_mode(self, file_index, doc_index): """Should generate relative paths for doc links.""" - content = "See [Development](.md)" - doc_path = REPO_ROOT / "docs/concepts/test.md" + content = "See [Blueprints](.md)" + doc_path = REPO_ROOT / "docs/api/test.md" new_content, _changes, errors = process_markdown( content,