Skip to content

Add receipt / log reads to cryptosim#3081

Merged
jewei1997 merged 31 commits intomainfrom
cryptosim-receipt-gen-reads
Apr 9, 2026
Merged

Add receipt / log reads to cryptosim#3081
jewei1997 merged 31 commits intomainfrom
cryptosim-receipt-gen-reads

Conversation

@jewei1997
Copy link
Copy Markdown
Contributor

@jewei1997 jewei1997 commented Mar 17, 2026

Describe your changes and provide context

This PR extends the cryptosim receipt benchmark from a write-only workload into a mixed read/write workload that exercises the production receipt store path more realistically. It adds concurrent receipt readers to RecieptStoreSimulator, uses deterministic synthetic tx hashes so readers can reconstruct lookup targets without storing per-tx state, and simulates both receipt-by-hash queries and eth_getLogs-style log filter reads with a configurable recency bias toward newer blocks.

To make that workload measurable, the PR adds receipt read metrics and dashboard panels for read latency, reads/sec, cache hit vs miss rate, found vs not found results, and log filter latency. It also adds an optional receipt read observer so cache hits/misses are recorded at the ledger-cache layer, updates the parquet receipt store to return ErrNotFound cleanly when there is no legacy KV fallback, and refreshes the sample receipt-store config to enable the new benchmark mode. Overall, this gives us a better way to evaluate receipt-store behavior under concurrent reads, writes, cache pressure, and pruning.

Testing performed to validate your change

Ran go test ./sei-db/ledger_db/receipt/...
Ran go test ./sei-db/state_db/bench/cryptosim/...
Added unit coverage for deterministic SyntheticTxHash generation
Added unit coverage for receipt cache hit/miss observer reporting

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 17, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedApr 8, 2026, 11:58 PM

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 17, 2026

Codecov Report

❌ Patch coverage is 69.35484% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.03%. Comparing base (289bc61) to head (7655181).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sei-db/ledger_db/receipt/cached_receipt_store.go 62.79% 13 Missing and 3 partials ⚠️
sei-db/ledger_db/receipt/parquet_store.go 33.33% 2 Missing ⚠️
sei-db/ledger_db/parquet/store.go 85.71% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #3081   +/-   ##
=======================================
  Coverage   59.02%   59.03%           
=======================================
  Files        2065     2065           
  Lines      169349   169405   +56     
=======================================
+ Hits        99960   100005   +45     
- Misses      60630    60641   +11     
  Partials     8759     8759           
Flag Coverage Δ
sei-chain-pr 75.85% <69.35%> (?)
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/config/receipt_config.go 76.19% <100.00%> (+0.58%) ⬆️
sei-db/ledger_db/parquet/reader.go 78.33% <100.00%> (+1.48%) ⬆️
sei-db/ledger_db/parquet/wal.go 76.11% <100.00%> (+0.73%) ⬆️
sei-db/ledger_db/receipt/receipt_store.go 74.74% <100.00%> (+0.25%) ⬆️
sei-db/ledger_db/parquet/store.go 68.91% <85.71%> (+0.46%) ⬆️
sei-db/ledger_db/receipt/parquet_store.go 68.16% <33.33%> (+0.74%) ⬆️
sei-db/ledger_db/receipt/cached_receipt_store.go 85.97% <62.79%> (-8.34%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jewei1997 jewei1997 changed the title cryptosim reads Add receipt / log reads to cryptosim Mar 18, 2026
Base automatically changed from cryptosim-receipt-gen to main March 18, 2026 20:28
}
}

func (r *txHashRing) Push(txHash common.Hash, blockNumber uint64, contractAddress common.Address) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, godocs are nice for exported functions

Comment on lines +118 to +120
// Cryptosim passes its metrics as a read observer so cache hits/misses are measured
// at the cache wrapper, which is the only layer that can distinguish them reliably.
store, err := receipt.NewReceiptStoreWithReadObserver(storeCfg, nil, metrics)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it would be useful to be able to intercept this information for the cache, we should discuss potential API. Doesn't sound like a very tricky feature to add.

}

func selectReceiptReadBlock(
rng *rand.Rand,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential hotspot, consider using canned random. (rand.Rand appears in a bunch of places in this file, comment applies to all use sites)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if there are rand.Rand features you'd like to have that aren't in canned random, it should be fairly straight forward to add additional convenience methods to canned random that return whatever data type is convenient.

@jewei1997 jewei1997 changed the base branch from main to cache-first-filter-logs March 30, 2026 18:51
@jewei1997 jewei1997 force-pushed the cryptosim-receipt-gen-reads branch from 38fddce to 9c1a5b6 Compare March 30, 2026 19:25
Add a lightweight pebble-backed index mapping tx_hash -> block_number
that narrows GetReceiptByTxHash to the single parquet file containing
the target block, instead of scanning all files. Falls back to full
scan for tx hashes not yet in the index.

- New TxIndex type with SetBatch, GetBlockNumber, PruneBefore, Close
- Store wires tx index into WriteReceipts, GetReceiptByTxHash, Close,
  SimulateCrash, and periodic pruning
- Reader gains GetReceiptByTxHashInBlock for targeted single-file query
- Unit tests for TxIndex ops and integration tests for Store/Reader
Extend cryptosim from write-only to mixed read/write workload: add
concurrent receipt-by-hash and eth_getLogs-style log filter readers,
deterministic synthetic tx hashes for stateless lookups, recency-biased
block selection, read metrics/dashboard panels, receipt read observer
for cache hit/miss tracking, and clean ErrNotFound from parquet store.

Made-with: Cursor
…degradation

Instruments the ledger cache layer to measure:
- FilterLogs cache scan duration (separate from backend)
- GetReceipt cache lookup duration (includes clone cost)
- Total log and receipt counts across cache chunks

Adds corresponding Grafana dashboard panels.

Made-with: Cursor
EstimatedReceiptCacheWindowBlocks (1000 blocks) overshoots the actual
cache coverage after rotation (~500 blocks), causing ~70% of cache-mode
log filter reads to miss and fall through to DuckDB.

Made-with: Cursor
…index

Two performance improvements for duckdb-only reads:

1. GetLogs now skips parquet files whose block range is entirely before
   FromBlock, matching the existing ToBlock pruning. This prevents DuckDB
   from scanning irrelevant historical files.

2. Add a lightweight pebble-backed tx_hash -> block_number index that
   narrows GetReceiptByTxHash to the single parquet file containing the
   target block, instead of scanning all files. Falls back to full scan
   for tx hashes not yet in the index.

Made-with: Cursor
Base automatically changed from cache-first-filter-logs to main April 3, 2026 20:41
@jewei1997 jewei1997 force-pushed the cryptosim-receipt-gen-reads branch from 7f2e2db to 90505eb Compare April 7, 2026 16:20
@jewei1997 jewei1997 enabled auto-merge April 8, 2026 23:58
@jewei1997 jewei1997 added this pull request to the merge queue Apr 9, 2026
Merged via the queue into main with commit ebb3832 Apr 9, 2026
39 checks passed
@jewei1997 jewei1997 deleted the cryptosim-receipt-gen-reads branch April 9, 2026 00:25
yzang2019 added a commit that referenced this pull request Apr 9, 2026
* main:
  Add receipt / log reads to cryptosim (#3081)
  persist blocks and FullCommitQCs in data layer via WAL (CON-231) (#3126)
  Update Changelog in prep to cut v6.4.1 (#3213)
  fix(sei-tendermint): resolve staticcheck warnings (#3207)
  Add historical state offload stream hook (#3183)
  feat: wire autobahn config propagation from top-level to GigaRouter (CON-232) (#3194)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants