Bonsai Archive with State Proofs #8918

matthew1001 · 2025-07-04T10:06:54Z

PR description

(Replaces previous draft PR #8669 after having completed a fairly involved merge.)

Outstanding TODOs:

Decide whether we want X_BONSAI_ARCHIVE_PROOFS and X_BONSAI_ARCHIVE_PROOFS as discrete DB format types, or whether we should just have proofs enabled for all bonsai archive DBs
- Discord discussion (https://discord.com/channels/905194001349627914/1391784271446020116/1391784404896059483) landed on keeping these as discrete config options
Make checkpoint interval a persisted attribute of the DB storage metadata
- I don't think we can change the interval at which checkpoints are taken. At least, it's certainly not MVP. It should either be fully supported that the checkpoint interval can change, or one it is set on a fresh DB it is set in stone. For this PR I'm proposing the latter.
Agree on use of flat naming convention in the classes
- In-person zoom with @garyschulte and @matkt determined that there may be optimizations to be made to the existing implementation, perhaps by combining the current approach with hash-based lookups, but that the current PR was valid in its current state and open to improvements in future PRs. Hence the current "flat" lookup approach is being retained for the time being
Remaining PR comments

This PR follows on from the first Bonsai Archive PR and enhances it to provide full state proofs for Bonsai Archive state.

It introduces a new experimental data storage format X_BONSAI_ARCHIVE_PROOFS
- I would propose that once proven out in experimental mode, the 2 X_BONSAI_ARCHIVE and X_BONSAI_ARCHIVE_PROOFS options are merged into a single BONSAI_ARCHIVE data storage format. Currently I think it will be useful to have separate options while in experimental to make it easier to recreate any issues with and without the state proof behaviour.
I have added a page to the Besu wiki which contains the PDF I presented to the Besu community call. The PDF should provide a reasonable amount of design detail, including key/value formats used.

The aim of this feature is to provide feature parity with FOREST DB and allow us to finally start the process of removing FOREST DB from Besu.

Testing

Aside from the updated tests in the PR which exercise the new flat DB format by adding X_BONSAI_ARCHIVE_PROOFS to various existing tests, I have run a number of QBFT chains using a combination of FOREST, BONSAI, and X_BONSAI ARCHIVE_PROOFS nodes and exercised a variety of state update & state proof requests.

I have configured a node to sync with Ethereum mainnet and it is currently at block 7m without issues:

In addition to syncing with mainnet, I've created 2 test scripts which exercise eth_getTransactionCount, eth_getStorageAt, and eth_getProof for known accounts/states in the first 3m blocks. The eth_getProof script uses proofs obtained from a FOREST node synced up to 3+m blocks, and then uses eth_getProof against the BONSAI archive node to check that the proofs returned by BONSAI match those returned by FOREST.

Script for eth_getProof testing on mainnet

#!/bin/bash

nonceResponseMatches () {
echo -n "Checking nonce == $2 for account $3, block $4 - "
if [[ "${1^^}" == "${2^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $1 != $2"
  exit 1
fi
}

balanceResponseMatches () {
echo -n "Checking balance == $2 for account $3, block $4 - "
if [[ "${1^^}" == "${2^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $1 != $2"
  exit 1
fi
}

storageResponseMatches () {
echo -n "Checking storage slot $1 == $3 for account $4, block $5 - "
if [[ "${2^^}" == "${3^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $2 != $3"
  exit 1
fi
}

proofResponseMatches () {
echo -n "Checking proof == $2 for account $3, block $4 - "
if [[ "${1^^}" == "${2^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $1 != $2"
  exit 1
fi
}

# Retrive the transaction count for a number of different accounts in the first 1,000,000 blocks of Ethereum L1
# Some of these have been specfically selected for accounts that change several times in a block, or that change in several contiguous blocks.

echo 
echo State proof check for Block 150002, account 0x32Be343B94f860124dC4fEe278FDCBD38C102D88
echo
ACCOUNT="0x32Be343B94f860124dC4fEe278FDCBD38C102D88"
BLOCK="0x249F2"
PROOF=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.accountProof`
PROOF=`echo $PROOF | sed "s/\"/'/g"`
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.nonce`
nonceResponseMatches $NONCE "0x805" $ACCOUNT $BLOCK
BALANCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.balance`
balanceResponseMatches $BALANCE "0x8bf9c8d1322d1741740" $ACCOUNT $BLOCK
EXPECTEDPROOF=`echo '[ "0xf90211a06d095eaaeacdfffd075684f0578cb572fc88606acd332f7c613de8152a6321a2a0979a68d3084102613fd289c4394e6474b692986bcd03705273f1cca529cb1571a0f54ba75f04abfa8c99886369a13562fb1ccfab979c1332404f1a1039f5603b12a03c117529a1c47692faae3f7d6cfa1d01b1a4345e943609aac9e3123721f1a18fa0d591655460aa3f7fcc03f45e0f0a512a3954aa9cf5105345aa70fd24052c0c96a02b0ef78ba5968e66764694429397e5b52a2e86d8dec695cf0c51cea0e776ebdea03a280eb9719f1b9cc58d59e42ee20f5ee4826922ed73451ea7b265dad0c00c21a02cf89409a859fc6f72aebf9f56d2d4ca8991bf07255442eee7a57e18e7712d35a04b3526fa5187227bda323ff89f6347cfa9e3906b26b101a910c0e74b49f44571a04fc9d79b109dce78daddbd8de09ecbcbcfb404b05d94ec3bb24e670378eae21ba0a37eb851d5af0bedd13eb9262491f300f688bf4547b2e1bc9f5c522c3aa36673a0e21abca7946f0cd6750f41c4994ddbcda5e5702d29391f554dcf290e493f6335a027c1cbda50110c884482266ec4c449691e835b1761da3cfb772db6ee0664f3a4a06aa66a16f28af0b1caca724bc2a3b85896254e08c070c751715b5fc1e44b3150a0aef676628e901f0a8b09782ff39fc2721bb69d8f0a58228393f709dd2e2bfef4a0cdcc4b6cd4ab9bcca73ca22836e19430efead4146e77851add1e9428c867dbdd80", "0xf90211a001883aa7e1f64a4ff0da04f7aa5f219d3ccc56e5fb351038df047d6984f0d956a0272952891fec9bb5663025ea019e3cb71b78362f08bc0d864ca196d17bd3a672a076c4c9fbb7d90cc93601f8c79c30afd037940a257062b3441168a91b6a00473ea08de19c85de6ff38585b5d411f37bba44fb90b5609d5e5ea9509e5ca1b5369b6aa0fb5f0a1a6ced0c97d162fda76900798cadeb14e24590b5e55c5ecb9cc5074f48a0c8a1a880604a20721048d5649030f05093b7b5b52b7fd73f5193b6984d385640a063138db8e2ddf2ac2d98743c9f82624a27b1b51b97a2a1069da4db9055192fbba0a39d481d2355f0451e725b41f0b7641816eccc9dc9e719a3c09765b383609d11a071c1994df821d3a718db10baaecf11a994ce2f32cac10a91ce69c7b5e71136c8a05eb07a4ab7eed93c614a9aca3c1afabbf06eb5dc9c8bc4bbdf5f786b647a3461a01fb3df47069f7eb2640f612343f642b44415094a7e9db11b6bbfc5d9dda4754da0dd76b20038570047e65010f35dc68177021c005d818220bfe1ed10694294d56ba06968f0ff479395925820fc71c7ba5cfeed7ca5f1bbf5566990a045c834c38e14a0b5660ae3d933376c9aa107c864ec2724ea8ecfc3e7a943c198b3486b5b9e358ea0585bb49b9e23c53ed785e27156e86adbb8c913780465700f56d431e0bf0beb04a0cb964717598a937e92f9d6ff514fa6acbf86e88de1333d7c9f1798251c21114880", "0xf90211a0022f94fbe307cbb1e956286f7486e96f668a040b618df1b35b3f1b89e79ae622a091de5983a08e4ce81ffd92f5f3c86973b26c475388fe5a6c3ff473ba37c82d66a01eb1892d9ed86ef3a2f2ce53a0735b1b34c367993e79717e20e9dcd05b235ed2a08f77ab9901240950440678e6d97685a648b3837a004c3914d6a9100465e95875a0685c883719c752347ad096c5df5106dafe3ea0bb71ca7bc123de4bd5fdd71200a0427e40d6e6955dc4e05f009365a06287cf64a87dc55d3105d3cde13ec8126f6ea033c430dc41783155398f4dafa602de2292be02071255f68d417ac84f3cf2392ea03cf7ea6ce6e7b60e886baab8a24aeb83ec56b1b8e5d1faac7dc11c3d8c7cab4fa0f5ebceebf282e4286290bff1d1335b9a0aaa5a5e856e64cb0579ef852b3980e7a06e8e92a67e695493f01f428932f224c0ff6268fcea2f70cdd7ea131d083df74ea0620fe1b7bf2396bd3adcdb6133343193b1319285199b00e88e5fe030b79d050ea08e7eca44e7a52bab6aea332dbbca859542b0f1f306dec9663c685e2b29ea1d76a09b86f98ca6c0baa062d9092c7863bcff0c9f2c6c1a801e24b0bba32e1df475fca0c658bcafc8219b1f779e4f287ad62eabf99cecf2e182d4e1a18ecf5bfaaab026a0e0769df0ef25c8bf2501e4a40369a552582a215357c15ee6b030901b1240e9aca0c52be2cf694dd32f923cd2d2e1270b21d8887e33f34bdf05c69a8f3446a3c00380", "0xf89180a053ddec1f33fa7d5372fc91725332a72af561989fd31982e4b8b2e9ce9809c406808080a07abdc05b292174e3e9de48d893ca5e7d721ba05884fe4e9b3cd59dca47359903a0ba4ba8e064b1fc2273267fd1942f211233edd61388c6c5644d561dd0f2a491518080808080808080a010d3c12c3e1efce1c97c3530a568ebfb38479624a66e6f8cab462b9a90b0b43480", "0xf8749f209f05d2dac481a991ac6466e42d76cfcbfe9d59681d3f255cc95c3b87cbe0b852f8508208058a08bf9c8d1322d1741740a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470" ]' | sed "s/\"/'/g"`
proofResponseMatches "$PROOF" "$EXPECTEDPROOF" $ACCOUNT $BLOCK

echo 
echo State proof check for Block 150003, account 0x32Be343B94f860124dC4fEe278FDCBD38C102D88
echo
BLOCK="0x249F3"
PROOF=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.accountProof`
PROOF=`echo $PROOF | sed "s/\"/'/g"`
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.nonce`
nonceResponseMatches $NONCE "0x807" $ACCOUNT $BLOCK
BALANCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.balance`
balanceResponseMatches $BALANCE "0x87590bb2b291a6d1240" $ACCOUNT $BLOCK
EXPECTEDPROOF=`echo '[ "0xf90211a06d095eaaeacdfffd075684f0578cb572fc88606acd332f7c613de8152a6321a2a0979a68d3084102613fd289c4394e6474b692986bcd03705273f1cca529cb1571a0f54ba75f04abfa8c99886369a13562fb1ccfab979c1332404f1a1039f5603b12a03c117529a1c47692faae3f7d6cfa1d01b1a4345e943609aac9e3123721f1a18fa00c6adfb8616db3857cb955fab23c6108e1f7ecb578a7456c8f8d078f9df31864a02b0ef78ba5968e66764694429397e5b52a2e86d8dec695cf0c51cea0e776ebdea03a280eb9719f1b9cc58d59e42ee20f5ee4826922ed73451ea7b265dad0c00c21a02cf89409a859fc6f72aebf9f56d2d4ca8991bf07255442eee7a57e18e7712d35a04b3526fa5187227bda323ff89f6347cfa9e3906b26b101a910c0e74b49f44571a0281cbb03a1415785a4df047741da8276d2e61b94d69fcce846697243993da320a0a37eb851d5af0bedd13eb9262491f300f688bf4547b2e1bc9f5c522c3aa36673a0e21abca7946f0cd6750f41c4994ddbcda5e5702d29391f554dcf290e493f6335a027c1cbda50110c884482266ec4c449691e835b1761da3cfb772db6ee0664f3a4a04de04af90a2179450e00b67ad50d33057629fb9f5cc8db32bd871c0b7b1177eda0aef676628e901f0a8b09782ff39fc2721bb69d8f0a58228393f709dd2e2bfef4a0a8ce2e50d3902840eec1ccb52d2e0e139fd1b45731f7740244eb5ab192d9b24d80", "0xf90211a001883aa7e1f64a4ff0da04f7aa5f219d3ccc56e5fb351038df047d6984f0d956a0272952891fec9bb5663025ea019e3cb71b78362f08bc0d864ca196d17bd3a672a076c4c9fbb7d90cc93601f8c79c30afd037940a257062b3441168a91b6a00473ea08de19c85de6ff38585b5d411f37bba44fb90b5609d5e5ea9509e5ca1b5369b6aa0fb5f0a1a6ced0c97d162fda76900798cadeb14e24590b5e55c5ecb9cc5074f48a0c8a1a880604a20721048d5649030f05093b7b5b52b7fd73f5193b6984d385640a063138db8e2ddf2ac2d98743c9f82624a27b1b51b97a2a1069da4db9055192fbba0a39d481d2355f0451e725b41f0b7641816eccc9dc9e719a3c09765b383609d11a071c1994df821d3a718db10baaecf11a994ce2f32cac10a91ce69c7b5e71136c8a05eb07a4ab7eed93c614a9aca3c1afabbf06eb5dc9c8bc4bbdf5f786b647a3461a01fb3df47069f7eb2640f612343f642b44415094a7e9db11b6bbfc5d9dda4754da0dd76b20038570047e65010f35dc68177021c005d818220bfe1ed10694294d56ba06968f0ff479395925820fc71c7ba5cfeed7ca5f1bbf5566990a045c834c38e14a0b5660ae3d933376c9aa107c864ec2724ea8ecfc3e7a943c198b3486b5b9e358ea0585bb49b9e23c53ed785e27156e86adbb8c913780465700f56d431e0bf0beb04a07f9a4cfe156c4d3f7598b72007fc4255c11c1f299614215048843706f8e63a8c80", "0xf90211a0022f94fbe307cbb1e956286f7486e96f668a040b618df1b35b3f1b89e79ae622a0d94d9397680e42b81f3465da02e570cc92aaa5ca80f5ff7fc40c8ad762ffeb91a01eb1892d9ed86ef3a2f2ce53a0735b1b34c367993e79717e20e9dcd05b235ed2a08f77ab9901240950440678e6d97685a648b3837a004c3914d6a9100465e95875a0685c883719c752347ad096c5df5106dafe3ea0bb71ca7bc123de4bd5fdd71200a0427e40d6e6955dc4e05f009365a06287cf64a87dc55d3105d3cde13ec8126f6ea033c430dc41783155398f4dafa602de2292be02071255f68d417ac84f3cf2392ea03cf7ea6ce6e7b60e886baab8a24aeb83ec56b1b8e5d1faac7dc11c3d8c7cab4fa0f5ebceebf282e4286290bff1d1335b9a0aaa5a5e856e64cb0579ef852b3980e7a06e8e92a67e695493f01f428932f224c0ff6268fcea2f70cdd7ea131d083df74ea0620fe1b7bf2396bd3adcdb6133343193b1319285199b00e88e5fe030b79d050ea08e7eca44e7a52bab6aea332dbbca859542b0f1f306dec9663c685e2b29ea1d76a09b86f98ca6c0baa062d9092c7863bcff0c9f2c6c1a801e24b0bba32e1df475fca0c658bcafc8219b1f779e4f287ad62eabf99cecf2e182d4e1a18ecf5bfaaab026a0e0769df0ef25c8bf2501e4a40369a552582a215357c15ee6b030901b1240e9aca0c52be2cf694dd32f923cd2d2e1270b21d8887e33f34bdf05c69a8f3446a3c00380", "0xf89180a053ddec1f33fa7d5372fc91725332a72af561989fd31982e4b8b2e9ce9809c406808080a0f8f540a67329a72e2174bd24e84b8829d5a742c3cab29b491c2e536b3bbcbc9ca0ba4ba8e064b1fc2273267fd1942f211233edd61388c6c5644d561dd0f2a491518080808080808080a010d3c12c3e1efce1c97c3530a568ebfb38479624a66e6f8cab462b9a90b0b43480", "0xf8749f209f05d2dac481a991ac6466e42d76cfcbfe9d59681d3f255cc95c3b87cbe0b852f8508208078a087590bb2b291a6d1240a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470" ]' | sed "s/\"/'/g"`
proofResponseMatches "$PROOF" "$EXPECTEDPROOF" $ACCOUNT $BLOCK

echo 
echo State proof check for Block 1000000, account 0x39fA8c5f2793459D6622857E7D9FbB4BD91766d3
echo
ACCOUNT="0x39fA8c5f2793459D6622857E7D9FbB4BD91766d3"
BLOCK="0xF4240"
PROOF=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.accountProof`
PROOF=`echo $PROOF | sed "s/\"/'/g"`
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.nonce`
nonceResponseMatches $NONCE "0x16" $ACCOUNT $BLOCK
BALANCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method": "eth_getProof","params": [ "'$ACCOUNT'", [], "'$BLOCK'"],"id": 1}' http://127.0.0.1:8545 | jq -r .result.balance`
balanceResponseMatches $BALANCE "0x15bac8c01b85fe4028" $ACCOUNT $BLOCK
EXPECTEDPROOF=`echo '[ "0xf90211a0c07e1b91223de332e1acdb7f51e34500bfc537de6a665034980459d1006605d7a07837218303eb727116ab34525441391c77fd65c8d6a3ec9d8b381822b83105a9a04f268ab2a53def380ada5b2912cb7217c50776ec90b773b53a794d9f423de013a02caeadaf17f2da451ba9bdf2ce9aca9ef14a990a0436156434c3f1f8d1f3c104a0c57482cbb52a96ee1fef320372d955a7a26bc63c8764cf8ccb174668669b441ba00baf463eb991fee008ab3f2e9191b8f18996d5ea3009f55a4cc8eaba9e0e3232a0310c4e4fa0a2fae15f82f855430662909cd36a105f4efbc82637a3c6e92f25b9a02121a1c9cb3ebd9130794ac5564122b3b1e6d29fec8e9d6e24398f084099cbdaa07b0918b6950b34282e6e0de398937ba77f5e4d1f2cc4fae5c840dcdb057df967a0e91917e14dab27f7793d1007aafb301f7c122f803f87c6e3ff382c17ad5f3b80a076c2dc94f6f18780bedabab5bea13e50e82ad48c6201de4e4a84e61cc4f10641a0182895c59c2705482eb700ad0b744a816da4c069091ebd3636cee60a53abb4f3a007afa3ee53d006bb9af6c8bc30e6abc02ab5a9d65edc2a62eb92fad2cb5cb19aa0ad87389f6f88f690811537e80f0f66b902109a2117693790957023ef76c55401a00ebef05bd0a10b5dedf2cbad85f50ce54fc9369b0afb5d1b7ecc3ee528da3f58a0931ea32e24b1747103be9909335432fe7315b23ecad64224ced0c5c948e6db8a80", "0xf90211a0c03e0c7cc5c0c6efc27834246c2ed8a59b057568ec21b4c7a4c94ce94d335436a0124683984ed87d8bad85bf0d5ba28509ee9d905eba60f978139515083b670f52a08ab95651f6405b5abd6756c306159425770b7163e0f10cbc1784b35b1a701930a04efc4743cad2db14e9c09e0bd205ba0a17de62b7f1a5843085529c2b4f166b75a05840981283f5e03d6f0077cfce4ea610e2e7c0eb8d3456c0648cf48b0c36fce0a072ae2d685e01d0ba25a7751e53c183c8a1e7d19264b14c81c77545aa483fc3a2a08fc6f340b05dc44fe58525604828031bcf227f61ba6a426d8e61b15876b7bddea0de3c91481c004312c774d43f009e1e2e747d52d07e30a5b4ebe6d4ae3a75f61ba0c703a7d2c3e12be1205a9b26b7db60837acbc63db7d26ba5e2c38f1a80d0677ea0bf3c789d8cccce4415c76bd72df374c0fa9b6fbbabffc58d2af99e12ef910257a06861b6a8a36817075dcc8f589966629287ce6f105e9ba92d0f65a9438250ae6ba012233fcf9b325f7d6411b8e997e61b766ada1ffdbbf88a1eae5ae16448ff6939a00d0d8800d6e13085494c0728cb576d49d2d70591257acdc10883cff1ae325bfca023ef7222872c1c6144b1841c3c2d8099774f1ba8d0e93eebd320805165a8d9c9a03a5bcdeb62e7b5a401d341438d7df8d478c0abada0357eff5e1acc475babecc9a047013c0d2e777a88fa3ef5ac5bb3d4dc1a6bc86272ccfb2387a5b5c3939a1f7880", "0xf90211a0a4d78502f7918e97c8c28fe271014726c3c2e191aea5a0a7bfe3c0b4ba496d33a0ebb3c34d26940a7f2137522ee2dc07b80596b4dffb0dec8a091d38f612f26a41a0e00a4a51c376943087b19ddad567da4e630c11f690b4d02d626ff109873ee419a0f2af0d9202ed43387023b43e8d1637e341e47b260afdef715e6fb28a8deef98ea0ba0e400f0c333955f65b91a4bc5554b2e20d7172bc8f70e5fb19e738ae022d20a0c6783bba73ff902e70ba7d0f26c29c49e9b1c6f94bdab9dcdd1eafb50bab5983a0f8ae64a9ca0fd6afebe0f2aea32726a3e13ea1501589ed7be73afb1a8f33b7dca0eb00787f4dac6a6473fa970a474197fb03505d0c9c2dbd8c21f19d69362a7fd7a08704d663a39ad9952da1ca0a642e10a52aeaa519d1d849a747b69d03567a0607a05678f5fd1caeada275bd3dd9c0ef3fb5eab96f2c8f81ac026b21e96e0e87305ca0815bfc20e92c88b409517320b2563fdb762e56d39aa8350bd99ae1b8289ce0cca08ecc7b9e54c44d99f5187cb8f0bdebafef7aaa49e3c09b16f612b557286a2617a03399e1628f1cb073038e877ae8ee14eea91cacddb946a4e124ddb602c5e277e5a02417a18fdfb3a43552ec260bc21e45f6ee03dbb3a52063593b03bbf8fa441263a0a38d0adf4e6cecbc0b0e09b921e90678cd365f068087a305a82f9c33c239ddc0a089feafebdd194fd78d1638e57d17a120c61a35710add3b2488dce66ed238e4ef80", "0xf9011180808080a06bd7f1dd56aac09dbe6b64752e0f3d686be5483f182fe8f51874b4b0fe86fb29a00aa8c9bfc05183cdee2bf7832bd6557d2dd040dafc056bf7288ea510f04afaf980a076d2a6dac166edd8418d90a2b9f1d05e290482d9683423ef0870b18abdfcceec80a04ab7910a13f05a5d9ff1b4946e402b91645246b123ea7e75b26085196d4c70c3a0d0406467e0536d2da0a773c1c4b21e57e1371510b3bc31616ca329cbe24843c18080a01ee655920fa772f97a7e8efa32e31de54f46168a03810ee7b3156465820d7feba044144bda0a998a5d36505d9ac5c4804f21522d9a288c3e4759101b4123992108a02f41006f70f4269034754873b225a7061a7896168e2b6a4f95d8c7e566dbec2780", "0xf8719f20ee3524a882a21ddf3073af6c0d567c6150d4a0ad787103a5282ade764735b84ff84d168915bac8c01b85fe4028a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470" ]' | sed "s/\"/'/g"`
proofResponseMatches "$PROOF" "$EXPECTEDPROOF" $ACCOUNT $BLOCK

exit 0

Script for testing historic account states on mainnet

#!/bin/bash

nonceResponseMatches () {
echo -n "Checking nonce == $2 for account $3, block $4 - "
if [[ "${1^^}" == "${2^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $1 != $2"
  exit 1
fi
}

storageResponseMatches () {
echo -n "Checking storage slot $1 == $3 for account $4, block $5 - "
if [[ "${2^^}" == "${3^^}" ]]; then
  echo OK
else
  echo "Unexpected JSON/RPC response. $2 != $3"
  exit 1
fi
}

# Retrive the transaction count for a number of different accounts in the first 1,000,000 blocks of Ethereum L1
# Some of these have been specfically selected for accounts that change several times in a block, or that change in several contiguous blocks.

# Block 150003 - 2 transactions from the same sender. Check nonce for blocks 150002, 150003, and 150004
ACCOUNT="0x32Be343B94f860124dC4fEe278FDCBD38C102D88"
BLOCK="0x249F2"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x805" $ACCOUNT $BLOCK
BLOCK="0x249F3"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x807" $ACCOUNT $BLOCK
BLOCK="0x249F4"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x807" $ACCOUNT $BLOCK

# Blocks  138719 and 138720 - transactions from the same sender in 2 contiguous blocks. Check blocks 138718, 138719, 138720 and 138721
ACCOUNT="0x1DCb8d1F0FCc8CbC8C2d76528E877F915e299fbE"
BLOCK="0x21DDE"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x59" $ACCOUNT $BLOCK
BLOCK="0x21DDF"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x5a" $ACCOUNT $BLOCK
BLOCK="0x21DE0"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x5b" $ACCOUNT $BLOCK
BLOCK="0x21DE1"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x5b" $ACCOUNT $BLOCK

# Some storage lookups for slots that are known to change
# Blocks 2018260, 2020000, 2300000 for a specific smart contract 
ACCOUNT="0x684282178b1d61164FEbCf9609cA195BeF9A33B5"

BLOCK="0x1ECBD4"
SLOT="0x5"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000003" $ACCOUNT $BLOCK
SLOT="0x7"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000001" $ACCOUNT $BLOCK
SLOT="0xa"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000000" $ACCOUNT $BLOCK

BLOCK="0x1ED2A0"
SLOT="0x5"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000004" $ACCOUNT $BLOCK
SLOT="0x7"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000001" $ACCOUNT $BLOCK
SLOT="0xa"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000000" $ACCOUNT $BLOCK

BLOCK="0x231860"
SLOT="0x5"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000005" $ACCOUNT $BLOCK
SLOT="0x7"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000001" $ACCOUNT $BLOCK
SLOT="0xa"
STORAGE_VAL=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getStorageAt","params":["'$ACCOUNT'","'$SLOT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
storageResponseMatches "$SLOT" $STORAGE_VAL "0x0000000000000000000000000000000000000000000000000000000000000001" $ACCOUNT $BLOCK

# Some other random checks for nonce on accounts at later blocks

# Block 2000000
ACCOUNT="0x32Be343B94f860124dC4fEe278FDCBD38C102D88"
BLOCK="0x1E8480"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x1EFC6" $ACCOUNT $BLOCK

# Block 3000000
ACCOUNT="0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8"
BLOCK="0x2DC6C0"
NONCE=`curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["'$ACCOUNT'","'$BLOCK'"],"id":1}' http://127.0.0.1:8545 | jq .result -r`
nonceResponseMatches $NONCE "0x10AA05" $ACCOUNT $BLOCK

exit 0

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matkt

first round of my review, added some comments and asked some questions

matkt · 2025-07-17T11:32:33Z

.../main/java/org/hyperledger/besu/tests/acceptance/dsl/node/configuration/BesuNodeFactory.java

-                    : storageFormat == DataStorageFormat.BONSAI
-                        ? DataStorageConfiguration.DEFAULT_BONSAI_CONFIG
-                        : DataStorageConfiguration.DEFAULT_BONSAI_ARCHIVE_CONFIG)
+                    : storageFormat == DataStorageFormat.X_BONSAI_ARCHIVE


maybe a switch will e better in this case ?

matkt · 2025-07-17T11:35:02Z

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/chain/Blockchain.java

+   * @param number The height of the block whose hash should be retrieved.
+   * @return The hash of the block at the given height.
+   */
+  Optional<Hash> getBlockHashByNumberSafe(long number);


what is the diff with getBlockHashByNumber ?

Actually I think this change can be removed. The intention was to add a synchronized version of getBlockHashByNumber() to resolve a race condition I was observing testing the state-proof code, similar to those fixed in #6344 and #6140. But I refactored the calling code and I don't think the synchronized version of getBlockHashByNumber() is currently required.

The getBlockHeaderSafe() I added is still required and called from BonsaiArchiveProofsWorldStateProvider so I'll leave that in.

I've tidied this up in the latest commits

matkt · 2025-07-17T11:37:06Z

...g/hyperledger/besu/ethereum/trie/pathbased/bonsai/BonsaiArchiveProofsWorldStateProvider.java

+  private Optional<BlockHeader> getCheckpointStateStartBlock(
+      final Blockchain blockchain, final Hash targetHash) {
+    long nearestCheckpointBlock =
+        (((blockchain.getBlockHeader(targetHash).get().getNumber() + trieNodeCheckpointInterval)


are we sure the block will be always here ?

Yeah that's a good point. I've updated it to tolerate the block not being found

...g/hyperledger/besu/ethereum/trie/pathbased/bonsai/BonsaiArchiveProofsWorldStateProvider.java

matkt · 2025-07-17T11:42:08Z

...g/hyperledger/besu/ethereum/trie/pathbased/bonsai/BonsaiArchiveProofsWorldStateProvider.java

+      final BlockHeader checkpointBlock,
+      final Hash targetBlockHash) {
+
+    // "Create" (fake) a world state representation at the next checkpoint block after the target


not sure to understand what is the fake worldstate

just setting the block; like that we are reading the right trie nodes ?

Yes I think I could clarify the comment wording here. "Fake" perhaps isn't the correct term. It's really just taking the current world state for the current chain head, and then starting the roll back process by asserting "this world state is for block X". The rollback logic interprets the asserted block number correctly when determining which trie logs it needs to roll. I think I will refactor the function names & comments to be clearer.

I've updated the function name in the latest commit, and reworded the comment.

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

matkt · 2025-07-17T11:52:26Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+    Bytes keyNearest =
+        calculateArchiveKeyWithMaxSuffix(
+            getStateArchiveContextForRead(storage),
+            Bytes.concatenate(accountHash, location).toArrayUnsafe());


maybe pushing Bytes.concatenate(accountHash, location).toArrayUnsafe()) into a byte[] field and use it for the size

Using tuweni is impacting performance so if we can avoid some call it's better

accountHash.size()

location.size()

Good thought - done

matkt · 2025-07-17T11:52:59Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+                            + 8)) // TODO - change for CONST length
+            .filter(
+                found ->
+                    Bytes.concatenate(accountHash, location).commonPrefixLength(found.key())


same here we are doing the concatenate multiple time and this one is really slow

matkt · 2025-07-17T11:59:44Z

...rg/hyperledger/besu/ethereum/trie/pathbased/common/provider/PathBasedWorldStateProvider.java

        return mapper.apply(
            worldStateProofProvider.getAccountProof(
-                ws.getWorldStateRootHash(), accountAddress, accountStorageKeys));
+                blockHeader.getStateRoot(), accountAddress, accountStorageKeys));


so the getWorldStateRootHash will not be the good one ?

I looked back at my change to see if there was a reason for this commit. I couldn't find one, so I've reverted the code to the original ws.getWorldStateRootHash() and re-tested with a short QBFT chain and I don't see any issues. So this change has now been removed.

...erledger/besu/ethereum/trie/pathbased/common/storage/PathBasedWorldStateKeyValueStorage.java

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

…ruption by misconfiguration Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 · 2025-11-04T13:53:37Z

Moving out of draft as it's only PR review comments I believe I have left to address

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 · 2025-11-07T14:39:05Z

I had my Ethereum L1 node stopped for a while (teku disk space ran out and I didn't notice). But I wanted to include the progress up to block 10m

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

jframe · 2025-12-19T03:41:45Z

I tried testing this on Linea Sepolia and it is failing on to import block 1. I get an error with the trie node not being found. So looks like there is an issue with genesis trie values.

This is the error I get:

{"@timestamp":"2025-12-17T08:06:57,205","level":"DEBUG","thread":"EthScheduler-Services-0","class":"PathBasedCachedWorldStorageManager","message":"found cached worldstate in cache for 0x65a64c825d7c13ce1bf077801d0b6b2a89853e19e4a89a5433d7d85d2102a20b","throwable":""}
{"@timestamp":"2025-12-17T08:07:01,587","level":"DEBUG","thread":"EthScheduler-Services-5 (importBlock)","class":"PathBasedCachedWorldStorageManager","message":"found cached worldstate in cache for 0x65a64c825d7c13ce1bf077801d0b6b2a89853e19e4a89a5433d7d85d2102a20b","throwable":""}
{"@timestamp":"2025-12-17T08:07:01,604","level":"DEBUG","thread":"EthScheduler-Services-5 (importBlock)","class":"PathBasedWorldState","message":"Persist world state for block Optional[BlockHeader{number=1, hash=0xcca982c33be23ec33d3a497deeff1727110e5fd298e4589dd9d78f7fcff7d0b1, parentHash=0x65a64c825d7c13ce1bf077801d0b6b2a89853e19e4a89a5433d7d85d2102a20b, ommersHash=0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, coinbase=0x0000000000000000000000000000000000000000, stateRoot=0x930d10f4e32de131381b76b22e11e3eccca54c6e4518d270d933d065b275efb3, transactionsRoot=0x60926104949721cae14c46d06646b3089eff3e85b06cb986eef5bd3d7879fe47, receiptsRoot=0xf78dfb743fbd92ade140711c8bbc542b5e307f0ab7984eff35d751969fe57efa, logsBloom=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty=0x0000000000000000000000000000000000000000000000000000000000000002, gasLimit=61000000, gasUsed=21000, timestamp=1709570965, extraData=0x0000000000000000000000000000000000000000000000000000000000000000bf3e45cb073fcc08e59743bd4b0ed8ac3573e1def48d053de079ca0c4ad5274b290d98165fa90a9dfc26455dbcdb585aef0c2cb1e51c72bb16a8be53738d4fc900, baseFee=0x0000000000000000000000000000000000000000000000000000000000000007, mixHashOrPrevRandao=0x0000000000000000000000000000000000000000000000000000000000000000, nonce=0, }]","throwable":""}
{"@timestamp":"2025-12-17T08:07:01,605","level":"INFO","thread":"EthScheduler-Services-5 (importBlock)","class":"DefaultSynchronizer","message":"
####################################################################################################
#                                                                                                  #
# Besu has identified a problem with its worldstate database.                                      #
# Your node will fetch the correct data from peers to repair the problem.                          #
# Starting the sync pipeline...                                                                    #
#                                                                                                  #
####################################################################################################","throwable":""}
{"@timestamp":"2025-12-17T08:07:01,606","level":"INFO","thread":"EthScheduler-Services-5 (importBlock)","class":"TransactionPoolFactory","message":"Disabling transaction handling during re-sync","throwable":""}
{"@timestamp":"2025-12-17T08:07:01,606","level":"WARN","thread":"EthScheduler-Services-5 (importBlock)","class":"BonsaiWorldStateProvider","message":"Invalid node for account 0x857dbfebd7d29a75ff06d1423c418da3b6e07292 at location 0x","throwable":""}

.../main/java/org/hyperledger/besu/tests/acceptance/dsl/node/configuration/BesuNodeFactory.java

jframe · 2025-12-17T06:09:20Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+                        + trieNodeCheckpointInterval
+                        + ")");
+              }
+              ;


nit: unnecessary semicolon

jframe · 2025-12-18T01:28:26Z

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/chain/DefaultBlockchain.java

@@ -371,6 +371,11 @@ public Optional<BlockHeader> getBlockHeader(final long blockNumber) {
    return blockchainStorage.getBlockHash(blockNumber).flatMap(this::getBlockHeader);
  }

+  @Override
+  public synchronized Optional<BlockHeader> getBlockHeaderSafe(final long blockNumber) {
+    return blockchainStorage.getBlockHash(blockNumber).flatMap(this::getBlockHeader);


I think this could just delegate to the non-safe one with this method using synchronized

Suggested change

return blockchainStorage.getBlockHash(blockNumber).flatMap(this::getBlockHeader);

return getBlockHeader(blockNumber);

jframe · 2025-12-18T01:59:22Z

CHANGELOG.md

  - SNAP sync will now download only headers for pre-checkpoint (pre-merge) blocks
  - `--snapsync-synchronizer-pre-checkpoint-headers-only-enabled` can be set to false to force SNAP sync to download pre-checkpoint (pre-merge) blocks
  - `--history-expiry-prune` can be used to enable online pruning of pre-checkpoint (pre-merge) blocks as well as modifying database garbage collection parameters to free up disk space from the pruned blocks
+- Experimental Bonsai Archive support enhanced to support `eth_proof` for historic blocks [#8918](https://github.com/hyperledger/besu/pull/8918)


should mention the bonsai archive proof format to make it clear it's not the bonsai archive data storage format that's being changed

jframe · 2025-12-18T02:13:42Z

...ger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchivePartialFlatDbStrategy.java

+import org.apache.tuweni.bytes.Bytes32;
+import org.apache.tuweni.rlp.RLP;
+
+public class BonsaiArchivePartialFlatDbStrategy extends BonsaiArchiveFlatDbStrategy {


Since the plan is to leave partial flat db strategy in for now, it would be good to include the metrics that we have in the BonsaiPartialFlatDbStrategy

jframe · 2025-12-19T02:17:01Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+      final NodeLoader nodeLoader,
+      final Bytes location,
+      final SegmentedKeyValueStorage storage) {
+    Optional<Bytes> accountFound;


This is a duplicate of getFlatAccountTrieNode can we just delegate to that

jframe · 2025-12-19T02:19:01Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+        storage
+            .getNearestBeforeMatchLength(TRIE_BRANCH_STORAGE, Bytes.of(keySuffixed))
+            .filter(found -> found.key().size() == (location.size() + KEY_SUFFIX_LENGTH))
+            .filter(found -> location.commonPrefixLength(found.key()) >= location.size());


This looks common between a lot of the methods can the logic be moved into a common method

jframe · 2025-12-19T02:33:49Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+          trieContext = Bytes.wrap(archiveRollingContext.get()).toLong();
+        } else {
+          trieContext =
+              (((Bytes.wrap(archiveContext.get()).toLong() + 1) / trieNodeCheckpointInterval)


I think this breaks if the trieNodeCheckpointInterval is 0 or 1 or possibly negative. I think this should be enforced in the config that is > 1 and add tests.

jframe · 2025-12-19T02:41:52Z

...dger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveProofsFlatDbStrategy.java

+
+        Optional<byte[]> archiveRollingContext =
+            storage.get(TRIE_BRANCH_STORAGE, ARCHIVE_PROOF_BLOCK_NUMBER_KEY);
+        if (archiveRollingContext.isPresent()) {


Relying on an temporary state being written to the database to get historical data isn't nice. Not sure there a better to solve this. Only thing I can think of it providing some context to the trie methods the archive proof strategy and retrieve the correct trie nodes. Curious to know if in your discussions with @matkt and @garyschulte whether you came up with any other better ideas?

jframe · 2025-12-19T03:34:40Z

...api/src/main/java/org/hyperledger/besu/plugin/services/storage/SegmentedKeyValueStorage.java

@@ -52,6 +52,20 @@ public interface SegmentedKeyValueStorage extends Closeable {
  Optional<NearestKeyValue> getNearestBefore(final SegmentIdentifier segmentIdentifier, Bytes key)
      throws StorageException;

+  /**
+   * Finds the key and corresponding value that is "nearest before" the specified key. "Nearest


Can you add to the description about the key length matching as well

…acceptance/dsl/node/configuration/BesuNodeFactory.java Co-authored-by: Jason Frame <jasonwframe@gmail.com> Signed-off-by: Matt Whitehead <matthew1001@hotmail.com>

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 changed the title ~~Merge WIP~~ Bonsai Archive with State Proofs Jul 4, 2025

matthew1001 mentioned this pull request Jul 4, 2025

Bonsai Archive (with state proofs) #8669

Closed

Bonsai archive state proofs

23eafc1

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 force-pushed the archive-proofs-rebase branch from a5b4d22 to 23eafc1 Compare July 4, 2025 10:09

matthew1001 added 4 commits July 4, 2025 11:32

Merge branch 'main' into archive-proofs-rebase

49a1216

Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>

Update plugin API hash

945c52b

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Unit test fix

b5fa260

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Merge branch 'main' into archive-proofs-rebase

fae5302

matkt reviewed Jul 17, 2025

View reviewed changes

matthew1001 added 2 commits August 11, 2025 11:45

Tidy up test code for choosing default storage format config

8523ffc

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Remove unused safe version of getBlockByHash. Spotless fixes.

f120446

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 force-pushed the archive-proofs-rebase branch from da1096e to f120446 Compare August 11, 2025 11:52

matthew1001 added 2 commits August 11, 2025 12:53

Refactor code & comments to make logic clearer

745d5c4

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Persist the checkpoint interval to trie log segment to prevent DB cor…

08c5b27

…ruption by misconfiguration Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 marked this pull request as ready for review November 4, 2025 13:53

Addressing PR comments

e0b22dc

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 force-pushed the archive-proofs-rebase branch from 1b1a335 to e0b22dc Compare November 5, 2025 10:42

matthew1001 added 2 commits November 7, 2025 13:01

Completing a couple of outstanding TODOs

44f0f94

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Revert to getWorldStateRootHash

4e50b84

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Last tidy up actions

b4f8de1

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 force-pushed the archive-proofs-rebase branch from e88078b to b4f8de1 Compare November 7, 2025 15:43

matthew1001 added 7 commits November 7, 2025 15:51

Merge branch 'main' into archive-proofs-rebase

7325250

Signed-off-by: Matt Whitehead <matthew.whitehead@kaleido.io>

Post-merge tidy up

7505791

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Remaining tidy up

2ff61c4

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Update plugin hash

8ab807f

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Test fix

6d4916c

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

More test fixes

9c93fcd

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Reset test rocks DB before each test

beb7718

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

matthew1001 added 3 commits November 10, 2025 16:47

Merge branch 'main' into archive-proofs-rebase

77b1ed0

Don't need to run both bonsai archive DB formats for peer tests

6f01def

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Merge branch 'main' into archive-proofs-rebase

88aa451

matthew1001 mentioned this pull request Nov 27, 2025

Bonsai archive genesis bug #9485

Merged

10 tasks

macfarla added this to RC 25.12.0 Dec 4, 2025

github-project-automation bot moved this to Backlog in RC 25.12.0 Dec 4, 2025

jframe force-pushed the archive-proofs-rebase branch from 9a4fbae to 88aa451 Compare December 17, 2025 07:41

jframe reviewed Dec 19, 2025

View reviewed changes

matthew1001 and others added 4 commits January 5, 2026 10:07

Update acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/…

b8239ef

…acceptance/dsl/node/configuration/BesuNodeFactory.java Co-authored-by: Jason Frame <jasonwframe@gmail.com> Signed-off-by: Matt Whitehead <matthew1001@hotmail.com>

Merge remote-tracking branch 'upstream/main' into archive-proofs-rebase

fe230bc

Fixes after merge

0b4ccfa

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

Plugin hash

75876b4

Signed-off-by: Matthew Whitehead <matthew.whitehead@kaleido.io>

	return blockchainStorage.getBlockHash(blockNumber).flatMap(this::getBlockHeader);
	return getBlockHeader(blockNumber);

Bonsai Archive with State Proofs #8918

Are you sure you want to change the base?

Bonsai Archive with State Proofs #8918

Uh oh!

Conversation

matthew1001 commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description

Testing

Uh oh!

matkt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matthew1001 commented Nov 4, 2025

Uh oh!

matthew1001 commented Nov 7, 2025

Uh oh!

jframe commented Dec 19, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

matthew1001 commented Jul 4, 2025 •

edited

Loading