Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/agents/docs/doclinks.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
When writing or editing markdown documentation, use `doclinks` tool to resolve file references.
When writing or editing markdown documentation, use the `doclinks` tool to resolve file references.

Full documentation if needed: [`utils/docs/doclinks.md`](/dimos/utils/docs/doclinks.md)

Expand Down
20 changes: 12 additions & 8 deletions docs/agents/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,31 @@
# Code Blocks

**All code blocks must be executable.**
Never write illustrative/pseudo code blocks.
Never write illustrative/pseudocode blocks.
If you're showing an API usage pattern, create a minimal working example that actually runs. This ensures documentation stays correct as the codebase evolves.

After writing a code block in your markdown file, you can run it by executing
`md-babel-py run document.md`
After writing a code block in your markdown file, you can run it by executing:
```bash
md-babel-py run document.md
```

more information on this tool is in [codeblocks](/docs/agents/docs_agent/codeblocks.md)
More information on this tool is in [codeblocks](/docs/agents/docs/codeblocks.md).


# Code or Docs Links

After adding a link to a doc run
After adding a link to a doc, run

`doclinks document.md`
```bash
doclinks document.md
```

### Code file references
```markdown
See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation.
```

After running doclinks, becomes:
After running doclinks, it becomes:
```markdown
See [`service/spec.py`](/dimos/protocol/service/spec.py) for the implementation.
```
Expand All @@ -48,7 +52,7 @@ Becomes:
See [Configuration](/docs/concepts/configuration.md) for more details.
```

More information on this in [doclinks](/docs/agents/docs_agent/doclinks.md)
More information on this is in [doclinks](/docs/agents/docs/doclinks.md).


# Pikchr
Expand Down
4 changes: 2 additions & 2 deletions docs/api/configuration.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Configuration

Dimos provides a `Configurable` base class, see [`service/spec.py`](/dimos/protocol/service/spec.py#L22).
Dimos provides a `Configurable` base class. See [`service/spec.py`](/dimos/protocol/service/spec.py#L22).

This allows using dataclasses to specify configuration structure and default values per module.

Expand Down Expand Up @@ -45,7 +45,7 @@ Error: Config.__init__() got an unexpected keyword argument 'something'

# Configurable Modules

[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids etc
[Modules]() inherit from `Configurable`, so all of the above applies. Module configs should inherit from `ModuleConfig` ([`core/module.py`](/dimos/core/module.py#L40)), which includes shared configuration for all modules like transport protocols, frame_ids, etc.

```python
from dataclasses import dataclass
Expand Down
10 changes: 5 additions & 5 deletions docs/api/sensor_streams/advanced_streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> **Prerequisite:** Read [ReactiveX Fundamentals](reactivex.md) first for Observable basics.

## Backpressure and parallel subscribers to hardware
## Backpressure and Parallel Subscribers to Hardware

In robotics, we deal with hardware that produces data at its own pace - a camera outputs 30fps whether you're ready or not. We can't tell the camera to slow down. And we often have multiple consumers: one module wants every frame for recording, another runs slow ML inference and only needs the latest frame.

Expand Down Expand Up @@ -43,7 +43,7 @@ from reactivex import operators as ops
from reactivex.scheduler import ThreadPoolScheduler
from dimos.utils.reactive import backpressure

# we need this scaffolding here, normally dimos handles this
# We need this scaffolding here. Normally DimOS handles this.
scheduler = ThreadPoolScheduler(max_workers=4)

# Simulate fast source
Expand Down Expand Up @@ -103,7 +103,7 @@ The `LATEST` strategy means: when the slow subscriber finishes processing, it ge

### Usage in modules

Most module streams offer backpressured observables
Most module streams offer backpressured observables.

```python session=bp
from dimos.core import Module, In
Expand All @@ -129,7 +129,7 @@ class MLModel(Module):

## Getting Values Synchronously

Sometimes you don't want a stream - you just want to call a function and get the latest value. We provide two approaches:
Sometimes you don't want a stream, you just want to call a function and get the latest value. We provide two approaches:

| | `getter_hot()` | `getter_cold()` |
|------------------|--------------------------------|----------------------------------|
Expand All @@ -154,7 +154,7 @@ source = rx.interval(0.1).pipe(ops.take(10))

get_val = getter_hot(source, timeout=5.0) # blocks until first message, with 5s timeout
# alternatively not to block (but get_val() might return None)
# get_val = getter_hot(source, nonblocking = True)
# get_val = getter_hot(source, nonblocking=True)

print("first call:", get_val()) # instant - value already there
time.sleep(0.35)
Expand Down
4 changes: 2 additions & 2 deletions docs/api/sensor_streams/quality_filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Qualities: [0.9]

For camera streams, we provide `sharpness_barrier` which uses the image's sharpness score.

Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Replay](/docs/old/testing_stream_reply.md) toolkit which provides access to recorded robot data:
Let's use real camera data from the Unitree Go2 robot to demonstrate. We use the [Sensor Replay](/docs/old/testing_stream_reply.md) toolkit, which provides access to recorded robot data:

```python session=qb
from dimos.utils.testing import TimedSensorReplay
Expand Down Expand Up @@ -177,7 +177,7 @@ plot_sharpness(input_frames, sharp_frames, '{output}')
<!--Result:-->
![output](assets/sharpness_graph.svg)

Let's request higher frequency
Let's request a higher frequency.

```python session=qb
sharp_frames = video_replay.stream(seek=5.0, duration=1.5, speed=1.0).pipe(
Expand Down
4 changes: 2 additions & 2 deletions docs/api/sensor_streams/reactivex.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Chain operators using `.pipe()`:
# Transform values: multiply by 2, then filter > 4
result = []

# we build another observable, it's passive until subscribe is called
# We build another observable. It's passive until `subscribe` is called.
observable = source.pipe(
ops.map(lambda x: x * 2),
ops.filter(lambda x: x > 4),
Expand Down Expand Up @@ -156,7 +156,7 @@ print("throttle_first() got:", results)
throttle_first() got: [0, 3, 6, 9]
```

### Difference between sample and throttle_first
### Difference Between `sample` and `throttle_first`

```python session=rx
# sample: takes LATEST value at each interval tick
Expand Down
2 changes: 1 addition & 1 deletion docs/api/sensor_streams/storage_replay.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ replay.stream(

A common pattern is creating replay-based connection stubs for testing without hardware. From [`robot/unitree/connection/go2.py`](/dimos/robot/unitree/connection/go2.py#L83):

This is a bit primitive, we'd like to write a higher order API for recording full module I/O for any module, but this is a work in progress atm.
This is a bit primitive. We'd like to write a higher-order API for recording full module I/O for any module, but this is a work in progress at the moment.


```python skip
Expand Down
18 changes: 9 additions & 9 deletions docs/api/sensor_streams/temporal_alignment.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ Out: box "(image, pointcloud)" rad 5px fit wid 170% ht 170%

## Basic Usage

Below we setup replay of real camera and lidar data from the Unitree Go2 robot, you can check if interested
Below we set up replay of real camera and lidar data from the Unitree Go2 robot. You can check it if you're interested.

<details>
<summary>Stream Setup</summary>

You can read more about [sensor storage here](storage_replay.md) and [LFS data store here](/docs/data.md)
You can read more about [sensor storage here](storage_replay.md) and [LFS data store here](/docs/data.md).

```python session=align no-result
from reactivex import Subject
Expand All @@ -47,7 +47,7 @@ import reactivex as rx
video_replay = TimedSensorReplay("unitree_go2_bigoffice/video")
lidar_replay = TimedSensorReplay("unitree_go2_bigoffice/lidar")

# this is a bit tricky, we find the first video frame timestamp, then add 2 seconds to it
# This is a bit tricky. We find the first video frame timestamp, then add 2 seconds to it.
seek_ts = video_replay.first_timestamp() + 2

# Lists to collect items as they flow through streams
Expand All @@ -70,9 +70,9 @@ lidar_stream = lidar_replay.stream(from_timestamp=seek_ts, duration=2.0).pipe(

</details>

Streams would normally come from an actual robot into your module via `IN` inputs, [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this.
Streams would normally come from an actual robot into your module via `IN` inputs. [`detection/module3D.py`](/dimos/perception/detection/module3D.py#L11) is a good example of this.

Assume we have them, let's align them.
Assume we have them. Let's align them.

```python session=align
# Align video (primary) with lidar (secondary)
Expand Down Expand Up @@ -166,7 +166,7 @@ plot_alignment_timeline(video_frames, lidar_scans, aligned_pairs, '{output}')
<!--Result:-->
![output](assets/alignment_timeline.png)

if we loosen up our match tolerance we might get multiple pairs matching the same lidar frame
If we loosen up our match tolerance, we might get multiple pairs matching the same lidar frame.

```python session=align
aligned_pairs = align_timestamped(
Expand Down Expand Up @@ -194,9 +194,9 @@ plot_alignment_timeline(video_frames, lidar_scans, aligned_pairs, '{output}')
<!--Result:-->
![output](assets/alignment_timeline2.png)

## We can combine frame alignment with a quality filter
## Combine Frame Alignment with a Quality Filter

more on [quality filtering here](quality_filter.md)
More on [quality filtering here](quality_filter.md).

```python session=align
from dimos.msgs.sensor_msgs.Image import Image, sharpness_barrier
Expand Down Expand Up @@ -239,7 +239,7 @@ plot_alignment_timeline(video_frames, lidar_scans, aligned_pairs, '{output}')
<!--Result:-->
![output](assets/alignment_timeline3.png)

We are very picky but data is high quality. best frame, with closest lidar match in this window.
We are very picky but data is high quality. Best frame, with closest lidar match in this window.

## How It Works

Expand Down
12 changes: 6 additions & 6 deletions docs/api/transforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,21 +279,21 @@ class PerceptionModule(Module):
"""Receives transforms and performs lookups."""

def start(self) -> None:
# this is just to init transforms system
# touching the property for the first time enables the system for this module.
# transform lookups normally happen in fast loops in IRL modules
# This is just to init the transforms system.
# Touching the property for the first time enables the system for this module.
# Transform lookups normally happen in fast loops in IRL modules.
_ = self.tf

@rpc
def lookup(self) -> None:

# will pretty print information on transforms in the buffer
# Will pretty-print information on transforms in the buffer
print(self.tf)

direct = self.tf.get("world", "base_link")
print(f"Direct: robot is at ({direct.translation.x}, {direct.translation.y})m in world\n")

# Chained lookup - automatically composes world->base->camera->optical
# Chained lookup - automatically composes world -> base -> camera -> optical
chained = self.tf.get("world", "camera_optical")
print(f"Chained: {chained}\n")

Expand Down Expand Up @@ -361,7 +361,7 @@ Transform tree:
```


You can also run `foxglove-studio-bridge` in the next terminal (binary provided by dimos and should be in your py env) and `foxglove-studio` to view these transforms in 3D (TODO we need to update this for rerun)
You can also run `foxglove-studio-bridge` in the next terminal (binary provided by DimOS and should be in your Python env) and `foxglove-studio` to view these transforms in 3D. (TODO we need to update this for rerun)

![transforms](assets/transforms.png)

Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/lcm.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

# LCM Messages

[LCM (Lightweight Communications and Marshalling)](https://github.com/lcm-proj/lcm) is a message passing system with bindings for many languages (C, C++, Python, Java, Lua, Go). While LCM includes a UDP multicast transport, its real power is the message definition format - classes that can encode themselves to compact binary representation.
[LCM (Lightweight Communications and Marshalling)](https://github.com/lcm-proj/lcm) is a message-passing system with bindings for many languages (C, C++, Python, Java, Lua, Go). While LCM includes a UDP multicast transport, its real power is the message definition format - classes that can encode themselves to a compact binary representation.

Dimos uses LCM message definitions for all inter-module communication. Because messages serialize to binary, they can be sent over any transport - not just LCM's UDP multicast, but also shared memory, Redis, WebSockets, or any other channel.

Expand Down Expand Up @@ -125,7 +125,7 @@ print(f"Memory transport: received {received[0]}")

# The LCM binary can also be sent raw over any byte-oriented channel
binary = msg.lcm_encode()
# send over websocket, redis, tcp, file, etc.
# send over WebSocket, Redis, TCP, file, etc.
decoded = Vector3.lcm_decode(binary)
print(f"Raw binary transport: decoded {decoded}")
```
Expand Down
28 changes: 14 additions & 14 deletions docs/concepts/modules.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@

# Dimos Modules

Modules are subsystems on a robot that operate autonomously and communicate to other subsystems using standardized messages.
Modules are subsystems on a robot that operate autonomously and communicate with other subsystems using standardized messages.

Some examples of modules are:

- Webcam (outputs image)
- Navigation (inputs a map and a target, outputs a path)
- Detection (takes an image and a vision model like YOLO, outputs a stream of detections)

Below is an example of a structure for controlling a robot. Black blocks represent modules and colored lines are connections and message types. It's okay if this doesn't make sense now, it will by the end of this document.
Below is an example of a structure for controlling a robot. Black blocks represent modules, and colored lines are connections and message types. It's okay if this doesn't make sense now. It will by the end of this document.

```python output=assets/go2_nav.svg
from dimos.core.introspection import to_svg
Expand All @@ -33,7 +33,7 @@ to_svg(CameraModule.module_info(), "assets/camera_module.svg")
<!--Result:-->
![output](assets/camera_module.svg)

We can always also print out Module I/O quickly into console via `.io()` call, we will do this from now on.
We can also print Module I/O quickly to the console via the `.io()` call. We will do this from now on.

```python session=camera_module_demo ansi=false
print(CameraModule.io())
Expand All @@ -53,14 +53,14 @@ print(CameraModule.io())
├─ Skill video_stream (stream=passive, reducer=latest_reducer, output=image)
```

We can see that camera module outputs two streams:
We can see that the camera module outputs two streams:

- `color_image` with [sensor_msgs.Image](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Image.html) type
- `camera_info` with [sensor_msgs.CameraInfo](https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) type

Offers two RPC calls, `start()` and `stop()`
It offers two RPC calls: `start()` and `stop()`.

As well as an agentic [Skill][skills.md] called `video_stream` (more about this later, in [Skills Tutorial][skills.md])
As well as an agentic [Skill](skills.md) called `video_stream` (more about this later, in [Skills Tutorial](skills.md)).

We can start this module and explore the output of its streams in real time (this will use your webcam).

Expand All @@ -69,7 +69,7 @@ import time

camera = CameraModule()
camera.start()
# now this module runs in our main loop in a thread. we can observe it's outputs
# Now this module runs in our main loop in a thread. We can observe its outputs.

print(camera.color_image)

Expand Down Expand Up @@ -122,9 +122,9 @@ print(Detection2DModule.io())

TODO: add easy way to print config

looks like detector just needs an image input, outputs some sort of detection and annotation messages, let's connect it to a camera.
Looks like the detector just needs an image input and outputs some sort of detection and annotation messages. Let's connect it to a camera.

```pythonx ansi=false
```python ansi=false
import time
from dimos.perception.detection.module2D import Detection2DModule, Config
from dimos.hardware.sensors.camera.module import CameraModule
Expand Down Expand Up @@ -153,17 +153,17 @@ Detection(Person(1))

## Distributed Execution

As we build module structures, very quickly we'll want to utilize all cores on the machine (which python doesn't allow as a single process), and potentially distribute modules across machines or even internet.
As we build module structures, we'll quickly want to utilize all cores on the machine (which Python doesn't allow as a single process) and potentially distribute modules across machines or even the internet.

For this we use `dimos.core` and dimos transport protocols.
For this, we use `dimos.core` and DimOS transport protocols.

Defining message exchange protocol and message types also gives us an ability to write models in faster languages.
Defining message exchange protocols and message types also gives us the ability to write models in faster languages.

## Blueprints

Blueprint is a pre-defined structure of interconnected modules. You can include blueprints or modules in new blueprints
A blueprint is a predefined structure of interconnected modules. You can include blueprints or modules in new blueprints.

Basic unitree go2 blueprint looks like what we saw before,
A basic Unitree Go2 blueprint looks like what we saw before.

```python session=blueprints output=assets/go2_agentic.svg
from dimos.core.introspection import to_svg
Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/transports.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

Transports enable communication between [modules](modules.md) across process boundaries and networks. When modules run in different processes or on different machines, they need a transport layer to exchange messages.

While the interface is called "PubSub", transports aren't limited to traditional pub/sub services. A topic can be anything that identifies a communication channel - an IP address and port, a shared memory segment name, a file path, or a Redis channel. The abstraction is flexible enough to support any communication pattern that can publish and subscribe to named channels.
While the interface is called "PubSub", transports aren't limited to traditional pub-sub services. A topic can be anything that identifies a communication channel: an IP address and port, a shared memory segment name, a file path, or a Redis channel. The abstraction is flexible enough to support any communication pattern that can publish and subscribe to named channels.

## The PubSub Interface

Expand Down Expand Up @@ -64,7 +64,7 @@ Received 2 messages:
{'temperature': 23.0}
```

The full implementation is minimal - see [`memory.py`](/dimos/protocol/pubsub/memory.py) for the complete source.
The full implementation is minimal. See [`memory.py`](/dimos/protocol/pubsub/memory.py) for the complete source.

## Available Transports

Expand Down
2 changes: 1 addition & 1 deletion docs/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Loaded pointcloud with 63672 points

Data files live in `data/` at the repo root. Large files are stored in `data/.lfs/` as `.tar.gz` archives tracked by Git LFS.

<details><summary>Diagon</summary>
<details><summary>Diagram</summary>

```diagon fold mode=Tree
data/
Expand Down
Loading