feat: competitive benchmark suite (story 12.3)#47
Merged
vieiralucas merged 20 commits intomainfrom Mar 7, 2026
Merged
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="bench/competitive/requirements.txt">
<violation number="1" location="bench/competitive/requirements.txt:2">
P2: Benchmark dependency versions are not pinned, which makes competitive results non-reproducible over time. Use exact versions (or a lockfile) for this benchmark suite.</violation>
</file>
<file name="bench/competitive/bench.py">
<violation number="1" location="bench/competitive/bench.py:608">
P1: NATS connection is closed too early, so later JetStream benchmark stages run on a closed client and fail.</violation>
</file>
<file name="_bmad-output/implementation-artifacts/12-3-competitive-benchmarks.md">
<violation number="1" location="_bmad-output/implementation-artifacts/12-3-competitive-benchmarks.md:29">
P2: AC #9 requires disk I/O metrics, but the defined/checked implementation only tracks `docker stats` resources, creating a completion mismatch in this story artifact.</violation>
</file>
<file name="bench/competitive/Makefile">
<violation number="1" location="bench/competitive/Makefile:35">
P2: Kafka readiness check can hang forever because it has no timeout/exit condition.</violation>
<violation number="2" location="bench/competitive/Makefile:41">
P2: RabbitMQ readiness check can hang indefinitely due to missing timeout.</violation>
<violation number="3" location="bench/competitive/Makefile:47">
P2: NATS readiness polling has no timeout, so failures become infinite waits.</violation>
</file>
<file name="bench/competitive/METHODOLOGY.md">
<violation number="1" location="bench/competitive/METHODOLOGY.md:109">
P2: The methodology states that all throughput benchmarks include warmup, but multi-producer throughput tests currently do not. This makes the benchmark documentation inconsistent with actual behavior.</violation>
</file>
<file name="bench/competitive/docker-compose.yml">
<violation number="1" location="bench/competitive/docker-compose.yml:32">
P1: `CLUSTER_ID` is set to an arbitrary string instead of a Kafka `random-uuid` cluster ID format, which can cause KRaft initialization failures.</violation>
<violation number="2" location="bench/competitive/docker-compose.yml:56">
P2: The NATS healthcheck depends on `wget`, but `nats:2.11` simple tags map to a scratch-based image where `wget` is not present, so health checks can fail continuously.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| 8. **Given** competitor configurations, **when** used, **then** they use recommended production settings (not default development settings). | ||
|
|
||
| 9. **Given** the benchmark runs, **when** results are collected, **then** resource utilization (CPU, memory, disk I/O) is included per broker during the benchmark. |
There was a problem hiding this comment.
P2: AC #9 requires disk I/O metrics, but the defined/checked implementation only tracks docker stats resources, creating a completion mismatch in this story artifact.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At _bmad-output/implementation-artifacts/12-3-competitive-benchmarks.md, line 29:
<comment>AC #9 requires disk I/O metrics, but the defined/checked implementation only tracks `docker stats` resources, creating a completion mismatch in this story artifact.</comment>
<file context>
@@ -0,0 +1,97 @@
+
+8. **Given** competitor configurations, **when** used, **then** they use recommended production settings (not default development settings).
+
+9. **Given** the benchmark runs, **when** results are collected, **then** resource utilization (CPU, memory, disk I/O) is included per broker during the benchmark.
+
+## Tasks / Subtasks
</file context>
|
|
||
| ### Warmup | ||
|
|
||
| All throughput benchmarks include a warmup period (default: 1 second) where messages are produced but not counted. This ensures the broker is in a steady state before measurement begins. |
There was a problem hiding this comment.
P2: The methodology states that all throughput benchmarks include warmup, but multi-producer throughput tests currently do not. This makes the benchmark documentation inconsistent with actual behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At bench/competitive/METHODOLOGY.md, line 109:
<comment>The methodology states that all throughput benchmarks include warmup, but multi-producer throughput tests currently do not. This makes the benchmark documentation inconsistent with actual behavior.</comment>
<file context>
@@ -0,0 +1,161 @@
+
+### Warmup
+
+All throughput benchmarks include a warmup period (default: 1 second) where messages are produced but not counted. This ensures the broker is in a steady state before measurement begins.
+
+### Multiple Runs
</file context>
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="bench/competitive/bench.py">
<violation number="1" location="bench/competitive/bench.py:261">
P2: Kafka multi-producer throughput includes warmup time in the denominator, which under-reports measured msg/s.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
vieiralucas
force-pushed
the
feat/12.2-ci-regression-detection
branch
2 times, most recently
from
12f48e9 to
eb51ad1
Compare
docker compose configs for kafka (kraft), rabbitmq (quorum queues), and nats (jetstream). python benchmark harness with throughput, latency, and lifecycle workloads. makefile orchestration for one-command execution. methodology documentation.
adds threaded multi-producer (3 concurrent producers) and fan-out (1 producer, 3 consumers) benchmarks for all three brokers. methodology doc updated with workload descriptions.
- fix nats connection closed too early before multi-producer/fan-out - use proper kafka kraft cluster id format - pin dependency versions for reproducibility - add timeout to makefile readiness checks (60s default) - add warmup to multi-producer benchmarks for consistency - document disk i/o limitation in methodology
the outer elapsed timer included warmup time, under-reporting throughput. each thread measures for exactly MEASURE_SECS, so use that directly.
vieiralucas
force-pushed
the
feat/12.3-competitive-benchmarks
branch
from
75c31b4 to
7425202
Compare
shared ci runners produce unreliable micro-benchmark results due to hardware variability. the compare step now uses continue-on-error so regressions are reported but don't block the pr. the bench-comment job also runs regardless of compare outcome.
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/bench-regression.yml">
<violation number="1" location=".github/workflows/bench-regression.yml:60">
P1: The regression comparison step is now non-blocking, so benchmark regressions can pass CI unnoticed.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| - name: Compare against baseline | ||
| if: github.event_name == 'pull_request' | ||
| continue-on-error: true |
There was a problem hiding this comment.
P1: The regression comparison step is now non-blocking, so benchmark regressions can pass CI unnoticed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/bench-regression.yml, line 60:
<comment>The regression comparison step is now non-blocking, so benchmark regressions can pass CI unnoticed.</comment>
<file context>
@@ -57,6 +57,7 @@ jobs:
- name: Compare against baseline
if: github.event_name == 'pull_request'
+ continue-on-error: true
run: |
if [ -f bench-baseline.json ]; then
</file context>
Suggested change
| continue-on-error: true | |
| continue-on-error: false |
Contributor
Benchmark Results (median of 3 runs)Commit:
|
replace python benchmark harness with a rust binary using native client libraries (rdkafka for kafka, lapin for rabbitmq, async-nats for nats). this ensures fair comparison by eliminating client language overhead — all brokers benchmarked with the same language and optimization level.
There was a problem hiding this comment.
1 issue found across 9 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="bench/competitive/METHODOLOGY.md">
<violation number="1" location="bench/competitive/METHODOLOGY.md:124">
P2: The methodology overstates runtime equivalence: Kafka uses threaded/librdkafka execution, while RabbitMQ/NATS use Tokio async tasks. Reword to avoid claiming all clients run in the same async runtime.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| - **NATS**: `async-nats` (official NATS Rust client) | ||
| - **Fila**: `fila-sdk` (native gRPC client) | ||
|
|
||
| This ensures the benchmark measures broker performance, not client language overhead. All clients run in the same Rust async runtime with equivalent optimization levels. |
There was a problem hiding this comment.
P2: The methodology overstates runtime equivalence: Kafka uses threaded/librdkafka execution, while RabbitMQ/NATS use Tokio async tasks. Reword to avoid claiming all clients run in the same async runtime.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At bench/competitive/METHODOLOGY.md, line 124:
<comment>The methodology overstates runtime equivalence: Kafka uses threaded/librdkafka execution, while RabbitMQ/NATS use Tokio async tasks. Reword to avoid claiming all clients run in the same async runtime.</comment>
<file context>
@@ -113,11 +113,15 @@ All throughput benchmarks include a warmup period (default: 1 second) where mess
+- **Fila**: `fila-sdk` (native gRPC client)
-For a strictly apples-to-apples comparison, one could benchmark all brokers using the same language. However, this would penalize Fila (whose Rust SDK is its primary client) or require maintaining Rust clients for Kafka/RabbitMQ/NATS.
+This ensures the benchmark measures broker performance, not client language overhead. All clients run in the same Rust async runtime with equivalent optimization levels.
### Hardware
</file context>
Suggested change
| This ensures the benchmark measures broker performance, not client language overhead. All clients run in the same Rust async runtime with equivalent optimization levels. | |
| This reduces client language overhead in the comparison, though client execution models still differ across libraries (for example, Kafka uses librdkafka/background threads while RabbitMQ and NATS use Tokio async tasks). |
rdkafka requires libcurl-dev which isn't available on ci runners. putting rdkafka, lapin, async-nats behind a "competitive" feature flag prevents cargo build --workspace from pulling them in. the bench-competitive binary uses required-features so it's only built when explicitly requested.
vieiralucas
force-pushed
the
feat/12.3-competitive-benchmarks
branch
from
836c1cf to
c09da51
Compare
verifies the bench-competitive binary compiles in ci by running cargo clippy with the competitive feature flag. installs libcurl-dev for rdkafka's cmake build.
vieiralucas
force-pushed
the
feat/12.3-competitive-benchmarks
branch
from
c09da51 to
c722509
Compare
There was a problem hiding this comment.
2 issues found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/bench-competitive.yml">
<violation number="1" location=".github/workflows/bench-competitive.yml:18">
P2: Pin `dtolnay/rust-toolchain` to a commit SHA instead of the mutable `stable` tag to avoid unreviewed action code changes in CI.</violation>
</file>
<file name="bench/competitive/Makefile">
<violation number="1" location="bench/competitive/Makefile:27">
P2: This shared `build` step now compiles `fila-server`/`fila-cli` for all broker benchmarks, even though only `bench-fila.sh` needs them and already builds them. Remove this duplicate compile from the common path to avoid unnecessary build time.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| - uses: dtolnay/rust-toolchain@stable |
There was a problem hiding this comment.
P2: Pin dtolnay/rust-toolchain to a commit SHA instead of the mutable stable tag to avoid unreviewed action code changes in CI.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/bench-competitive.yml, line 18:
<comment>Pin `dtolnay/rust-toolchain` to a commit SHA instead of the mutable `stable` tag to avoid unreviewed action code changes in CI.</comment>
<file context>
@@ -0,0 +1,39 @@
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+ - uses: dtolnay/rust-toolchain@stable
+ - uses: Swatinem/rust-cache@ad397744b0d591a723ab90405b7247fac0e6b8db # v2
+ - name: Install protoc
</file context>
The script cd's to REPO_ROOT before copying results, but OUTPUT_DIR was relative to bench/competitive/. Resolving to absolute path first.
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Contributor
Benchmark Results (median of 3 runs)Commit:
|
- fix nested tokio runtime panic: make kafka create_topic/cleanup_topic async instead of using block_on on current handle - fix kafka docker config: use PLAINTEXT://:9092 instead of 0.0.0.0 - use rabbitmq:3.13-management for docker compatibility - increase ready timeout to 120s for slower container startups - teardown each broker after benchmarking for resource isolation
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="bench/competitive/Makefile">
<violation number="1" location="bench/competitive/Makefile:47">
P2: Cleanup is not guaranteed on benchmark failure because `docker compose down -v` runs on a later recipe line that is skipped when the benchmark command exits non-zero.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| done; \ | ||
| if [ $$i -ge $(READY_TIMEOUT) ]; then echo "ERROR: Kafka not ready after $(READY_TIMEOUT)s"; exit 1; fi | ||
| $(BENCH_BIN) kafka $(RESULTS_DIR) | ||
| $(COMPOSE) down -v |
There was a problem hiding this comment.
P2: Cleanup is not guaranteed on benchmark failure because docker compose down -v runs on a later recipe line that is skipped when the benchmark command exits non-zero.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At bench/competitive/Makefile, line 47:
<comment>Cleanup is not guaranteed on benchmark failure because `docker compose down -v` runs on a later recipe line that is skipped when the benchmark command exits non-zero.</comment>
<file context>
@@ -44,6 +44,7 @@ bench-kafka: setup build
done; \
if [ $$i -ge $(READY_TIMEOUT) ]; then echo "ERROR: Kafka not ready after $(READY_TIMEOUT)s"; exit 1; fi
$(BENCH_BIN) kafka $(RESULTS_DIR)
+ $(COMPOSE) down -v
bench-rabbitmq: setup build
</file context>
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Contributor
Benchmark Results (median of 3 runs)Commit:
|
- Add -T flag to disable TTY allocation in exec commands (fixes CI where no pseudo-TTY is available) - Add container log dump on health check timeout for debugging - Increase RabbitMQ healthcheck retries and add start_period
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Docker Desktop for Mac VirtioFS creates .erlang.cookie with wrong permissions, causing RabbitMQ to crash on startup. Named volumes are managed entirely inside Docker's VM, bypassing VirtioFS.
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Docker Desktop for Mac VirtioFS causes .erlang.cookie eacces errors after heavy container I/O (e.g. Kafka benchmarks). Override entrypoint to explicitly create the cookie file with correct permissions before RabbitMQ starts.
Contributor
Benchmark Results (median of 3 runs)Commit:
|
- Remove --max_mem_store and --max_file_store from NATS command (not valid CLI flags in NATS 2.11, only config file options) - Use check_port_connectivity instead of ping for RabbitMQ readiness to ensure AMQP port is ready before benchmarking
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="bench/competitive/docker-compose.yml">
<violation number="1" location="bench/competitive/docker-compose.yml:70">
P2: Re-add explicit JetStream memory/file store limits; removing them makes NATS resource usage host-dependent and can skew or destabilize benchmark runs.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| ports: | ||
| - "4222:4222" | ||
| - "8222:8222" | ||
| command: ["--jetstream", "--store_dir", "/data", "-m", "8222"] |
There was a problem hiding this comment.
P2: Re-add explicit JetStream memory/file store limits; removing them makes NATS resource usage host-dependent and can skew or destabilize benchmark runs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At bench/competitive/docker-compose.yml, line 70:
<comment>Re-add explicit JetStream memory/file store limits; removing them makes NATS resource usage host-dependent and can skew or destabilize benchmark runs.</comment>
<file context>
@@ -67,12 +67,7 @@ services:
- --max_mem_store 256MB
- --max_file_store 1GB
- -m 8222
+ command: ["--jetstream", "--store_dir", "/data", "-m", "8222"]
healthcheck:
test: ["CMD-SHELL", "true"]
</file context>
Suggested change
| command: ["--jetstream", "--store_dir", "/data", "-m", "8222"] | |
| command: ["--jetstream", "--store_dir", "/data", "--max_mem_store", "256MB", "--max_file_store", "1GB", "-m", "8222"] |
Contributor
Benchmark Results (median of 3 runs)Commit:
|
- add mod fila to bench-competitive using FilaClient SDK with BenchServer - measure identical metrics: throughput (3 sizes), latency, lifecycle, multi-producer, and memory - remove fan-out benchmarks from kafka, rabbitmq, and nats modules (fila is a task queue, not pub-sub — unfair comparison) - update makefile bench-fila target to use bench-competitive binary - delete bench-fila.sh (no longer needed)
Contributor
Benchmark Results (median of 3 runs)Commit:
|
Contributor
Benchmark Results (median of 3 runs)Commit:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docker stats(CPU%, memory MB)make bench-competitiveruns all,make bench-{broker}runs individualTest plan
bench.py --helpparses correctlydocker compose config --quietvalidates compose filemake -n bench-competitivedry-run produces correct command sequencepy_compile)🤖 Generated with Claude Code
Summary by cubic
Adds a competitive benchmark suite comparing Fila with Kafka, RabbitMQ, and NATS using identical workloads and unified JSON reports. Rewrites the harness in Rust, integrates Fila in the same binary, adds one-command runs, production configs, and a CI workflow — completing Story 12.3.
New Features
Bug Fixes
Written for commit 1350fed. Summary will update on new commits.