Skip to content

feat: pin and Extract world state with adaptive paging#760

Merged
kant777 merged 1 commit intomainfrom
feature/eth_state_in_pages
Aug 18, 2025
Merged

feat: pin and Extract world state with adaptive paging#760
kant777 merged 1 commit intomainfrom
feature/eth_state_in_pages

Conversation

@kant777
Copy link
Copy Markdown
Contributor

@kant777 kant777 commented Aug 18, 2025

Pin debug state export to a fixed block with adaptive paging + storage handling

Summary

This PR updates the world-state → genesis exporter to reliably dump account balances, nonces, contract code, and (optionally) storage from an Erigon/Geth node—even under heavy state and strict RPC timeouts.

  • Pinned snapshot: Resolve latest once to a specific block number, print it, and use that same hex block for all calls.
  • Adaptive paging: Auto-reduce page sizes on timeouts for both accounts and storage.
  • Targeted storage export: Page storage per-contract only when requested, avoiding giant monolithic dumps.
  • Resilient merging: Preserve existing alloc entries and overlay the snapshot.

Why

Previous approaches could:

  • Time out on debug_dumpBlock or fat debug_accountRange calls.
  • Fail with missing trie node … loc: diff if the chain advanced mid-scan.
  • Over-fetch storage/code unnecessarily, causing instability on pruned nodes.

This PR fixes those by pinning the block and keeping RPC calls small and consistent.

What changed

  • Block pinning: latest → decimal block → single hex tag used for:
    • debug_accountRange(blockHex, start, n, true, true, true) (lightweight)
    • eth_getCode(addr, blockHex)
    • debug_storageRangeAt(blockHex, null, addr, startKey, n)
  • Adaptive pagination:
    • Accounts: halve --page-size on timeout down to --min-page-size.
    • Storage: halve --storage-page-size on timeout down to --storage-min-page-size.
  • Optional storage: Only fetched when --include-storage is set. Non-zero slots are included in alloc.storage.
  • Graceful degradation: If storage paging hits missing trie node (pruned node), we warn and continue with balances/nonces/code.
  • UX: Prints the exact block number being exported and a final output path.

Usage

python3 world_state4.py \
  --rpc http://<node>:8545 \
  --input-genesis initial_genesis.json \
  --output out_genesis.json \
  --include-code --include-storage \
  --page-size 2048 \
  --min-page-size 256 \
  --storage-page-size 2048 \
  --storage-min-page-size 128 \
  --rpc-timeout 300

@kant777 kant777 requested a review from harshsingh1002 August 18, 2025 19:48
@kant777 kant777 changed the title feat: pin and Extract world state at a fixed block with adaptive paging + storage handling feat pin and Extract world state at a fixed block with adaptive paging + storage handling Aug 18, 2025
@kant777 kant777 changed the title feat pin and Extract world state at a fixed block with adaptive paging + storage handling feat: pin and Extract world state at a fixed block with adaptive paging + storage handling Aug 18, 2025
@kant777 kant777 changed the title feat: pin and Extract world state at a fixed block with adaptive paging + storage handling feat: pin and Extract world state at a fixed block with adaptive paging Aug 18, 2025
@kant777 kant777 changed the title feat: pin and Extract world state at a fixed block with adaptive paging feat: pin and Extract world state with adaptive paging Aug 18, 2025
@kant777 kant777 merged commit ce66d3c into main Aug 18, 2025
5 of 10 checks passed
@kant777 kant777 deleted the feature/eth_state_in_pages branch August 18, 2025 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants