Skip to content

fix(grafana): $chain template variable empty in replay mode, profit panel renders $0 #424

@obchain

Description

@obchain

Symptom

During a fork-replay demo (charon replay --block 91323624 --borrower-file /tmp/seed.txt --hold-secs 1800), the Grafana dashboard's predicted-profit stat panel displayed $0 even though Prometheus had the correct value:

charon_executor_profit_usd_cents_sum{chain="bnb"} 3952448  # = $39,524.48

Other panels with no chain filter (queue depth, positions-by-bucket) populated correctly.

Root cause

The dashboard's \$chain template variable is sourced from:

label_values(charon_listener_blocks_received_total, chain)

charon_listener_blocks_received_total is incremented only by the listener subcommand on each new_heads WS event. In replay mode, no listener runs, so the metric is empty, and label_values() returns an empty set. With \$chain empty, every panel that filters chain=~"\$chain" matches no series.

The 13 \$chain consumers in the dashboard JSON include the predicted-profit stat panel (line 226), histogram heatmap, and several rate panels. All silently render $0 / empty during replay.

Fix

Change the variable source to a metric that emits in both listen and replay modes. charon_scanner_positions is a registered gauge (crates/charon-metrics/src/lib.rs:482-485) set unconditionally on every scanner tick in crates/charon-cli/src/main.rs:1958, 1998, 2026. Replay invokes the same run_block_pipeline (crates/charon-cli/src/main.rs:1833) that emits these gauges, so chain="bnb" is always present in the registry on first scrape.

Reproduction (before fix)

  1. cd deploy/compose && docker compose -f local-stack.yml up -d
  2. cargo run --release --bin charon -- --config config/fork.toml replay --block 91323624 --borrower-file <seed> --hold-secs 1800
  3. Open Grafana dashboard. Profit panel: $0.
  4. curl -s http://127.0.0.1:9091/metrics | grep profit_usd_cents_sum — metric is 3952448 (= $39 524.48).

Verification (after fix)

Same reproduction, profit panel shows $39 524.48.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions