Transport benchmarks + Raw ros transport

[leshy]

implements transport protocol benchmark suite

as well as raw ros transport (missing part to use this for unitree go2 etc is Dimos <-> ros msg translator, coming soon)

python -m pytest -svm tool dimos/protocol/pubsub/benchmark/test_benchmark.py

to run in docker:
./bin/dev python -m pytest -svm tool dimos/protocol/pubsub/benchmark/test_benchmark.py

[greptile-apps]

Greptile Summary

• Implements comprehensive transport benchmark suite to measure performance of different pubsub transport implementations (LCM, ROS, SharedMemory, Redis) across various message sizes with sophisticated visualization
• Adds new ROS 2 pubsub transport (RawROS) that enables direct integration with ROS ecosystems and provides foundation for future Unitree Go2 integration
• Enhances type safety across pubsub system by adding explicit EncodingT generic parameter to encoder mixins, making encoding format contracts more explicit

Important Files Changed

Filename	Overview
dimos/protocol/pubsub/benchmark/test_benchmark.py	New comprehensive benchmark test with threading, synchronization, and visualization; contains race condition where target_count is set before publish loop
dimos/protocol/pubsub/rospubsub.py	New ROS 2 transport implementation; has threading race conditions in executor/node access that could cause crashes during concurrent operations

Confidence score: 3/5

• This PR contains significant new functionality with threading complexity that requires careful review due to potential race conditions
• Score lowered due to race conditions in benchmark test (premature termination) and ROS transport (concurrent access to executor/node without proper synchronization)
• Pay close attention to threading implementation in both test_benchmark.py and rospubsub.py for race condition issues

Sequence Diagram

sequenceDiagram
    participant Test as "Test Framework"
    participant Benchmark as "test_throughput"
    participant PubSub as "Transport (LCM/ROS/SHM)"
    participant Callback as "Message Callback"
    participant Results as "BenchmarkResults"
    
    Test->>Benchmark: "Run benchmark test"
    Benchmark->>PubSub: "Create transport context"
    Benchmark->>PubSub: "Generate test message"
    Benchmark->>PubSub: "subscribe(topic, callback)"
    Benchmark->>Benchmark: "Sleep 0.1s (warmup)"
    
    loop "Publish Loop (1 second or MAX_MESSAGES)"
        Benchmark->>PubSub: "publish(topic, message)"
        PubSub->>Callback: "callback(message, topic)"
        Callback->>Callback: "Increment received_count"
        alt "All messages received"
            Callback->>Benchmark: "Set all_received event"
        end
    end
    
    Benchmark->>Benchmark: "Record publish_end time"
    Benchmark->>Benchmark: "Wait for remaining messages (RECEIVE_TIMEOUT)"
    Benchmark->>Benchmark: "Calculate latency and throughput"
    Benchmark->>Results: "add(BenchmarkResult)"
    Benchmark->>PubSub: "Stop transport context"
    
    Note over Test, Results: "After all tests complete"
    Results->>Results: "print_summary()"
    Results->>Results: "print_heatmap()"
    Results->>Results: "print_bandwidth_heatmap()"
    Results->>Results: "print_latency_heatmap()"

Loading

[greptile-apps]

_{7 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[leshy]

@greptileai review again plz

[greptile-apps]

_{11 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[paul-nechifor]

While this is the correct SI definition for GB, many still use GB to mean 1024**3. It could be less confusing to use 1024**3 while labeling it GiB. This way we don't confuse measurements.

[paul-nechifor]

I'm getting some packet loss for 5MB and 10MB. Do you get this too?

dimos/protocol/pubsub/benchmark/test_benchmark.py::test_throughput[LCM-5MB] Warning: LCM 5242880B: 11.7% message loss (326/369)
dimos/protocol/pubsub/benchmark/test_benchmark.py::test_throughput[UdpRaw-5MB] Warning: UdpRaw 5242880B: 60.1% message loss (901/2258)
dimos/protocol/pubsub/benchmark/test_benchmark.py::test_throughput[UdpRaw-10MB] Warning: UdpRaw 10485760B: 41.9% message loss (336/578)

Results for me:

[paul-nechifor]

I'm not sure if this is a realistic way to benchmark. By generating the message at the start of the loop, it's only measuring how long it takes to transport the message. That's fair, if we're only interested in measuring that aspect of message passing.

But real messages also take time to encode, and encodings can be cached giving unrealistic real-world expectations.

I've tried moving msggen to the publish loop, and the results are very different.

Also, I think we should be sending unique messages to make sure no caching is happening.

[leshy]

Actual encoding is happening at the transport layer and that's what we want to measure. I'm generating a source Image once (this generation I wanted outside of the loop itself) But an image is passed into the Transport layer, the transport specific encoding is executed (so lcm_encode, pickle, whatever ros does etc)

This is visible in the next PR #1041 in which I implement bytes only transports (to benchmark encoding costs, for example pickle vs lcm byte encode) and get much higher performance out of raw bytes transports

[paul-nechifor]

I've tried with something more sensible like 0.01, but it performs worse (in latency). Still, I don't think we should use 0, because that causes a busy loop wasting CPU even when no messages are being sent, right?

[leshy]

Yes, I'm not sure either, I will further review and optimize ros transport once it's ready to use, once message encoding is implemented (once we can pass dimos.msgs.* into it) here I was trying to give maximum possible edge to ROS since it was performing so badly, and it's used only in this test and nowhere else

[spomichter]

THis doesn't do anything any reason it needs to be a test

[leshy]

yeah we are using pytest as an easy way to specify new transports to test (via testdata.py) and a runner (runs a single test per transport/packet size combo)

and you can easily select which protocols to test via

python -m pytest -svk "lcm or ros" -m tool dimos/protocol/pubsub/benchmark/test_benchmark.py

this benchmark_results draws a graph for you for transports you chose (like images I've been attaching)

[leshy]

I'm getting some packet loss for 5MB and 10MB. Do you get this too?

great catch, actually Mac osx PR broke this, by changing service/lcm.py
TARGET_RMEM_SIZE = 2097152 # prev was 67108864
has to be 67108864

I'll make sure this is fixed in this PR, looks like I forgot to add the file - edit - pushed the fix