Skip to content

[pipeline] Per-block pipeline has no timeout; stall under real borrower load exceeds 3s BSC block time #174

@obchain

Description

@obchain

Refs #42

PR: feat(cli): wire scanner → router → builder → simulator pipeline (feat/17-cli-e2e-pipeline)
Commit: latest on feat/17-cli-e2e-pipeline
File: crates/charon-cli/src/main.rs (block event handler inside drain loop)

Problem:

BSC produces one block every ~3 seconds. The pipeline runs the following work per block, sequentially per liquidatable position:

  1. adapter.fetch_positions(borrowers) — N sequential eth_calls (one per borrower x vTokens)
  2. For each liquidatable position:
    a. adapter.get_liquidation_params() — 1 RPC call
    b. router.route() — 1 RPC call to Aave PoolDataProvider
    c. calculate_profit() — CPU only
    d. tx_builder.encode_calldata() — CPU only
    e. simulator.simulate() — 1 eth_call RPC call

For 50 borrowers and 5 liquidatable positions: ~50 + (5 x 3) = 65 sequential RPC calls per block, each with typical BSC RPC latency of 50-150ms. Worst case: 65 x 150ms = ~9.75 seconds per block, more than 3x the block time.

No tokio::time::timeout is imposed on the per-block tick. When the pipeline stalls, block events queue up in the mpsc channel. The listener's WS buffer fills, the WS connection drops and triggers the reconnect loop, causing missed blocks during recovery. This compounds the PR #32 finding 2 issue (blocking send stalls WS drain).

Impact: Under real load, the pipeline falls behind the chain head. Liquidation opportunities are processed N blocks late, by which time competing bots have already liquidated the position.

Fix:

  • Wrap the per-block handler in tokio::time::timeout(Duration::from_millis(2500), ...). Log warn! on timeout; do not panic.
  • Parallelize per-position work with JoinSet or futures::future::join_all.
  • Track follow-up for Multicall3 batching of fetch_positions to reduce N sequential calls to 1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglayer:rustRust crates (core / scanner / protocols / executor / cli)pr-reviewFindings from PR review processpriority:p1-coreCore MVP scopestatus:readyScoped and ready to pick up

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions