-
Notifications
You must be signed in to change notification settings - Fork 995
Enable decoding for large RPC requests #8877
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable decoding for large RPC requests #8877
Conversation
Max out StreamReadConstraints.maxStringLength for jackson decoding of request and parameters Create Engine API specific EngineJsonRpcParserHandler and EngineJsonRpcParameter containing modified ObjectMapper Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
...api/src/main/java/org/hyperledger/besu/ethereum/api/handlers/EngineJsonRpcParserHandler.java
Outdated
Show resolved
Hide resolved
|
When testing on mainnet, I go this error |
|
This PR needs at least the same level of testing as #8847 as it is changing how newPayload requests are deserialized |
...api/src/main/java/org/hyperledger/besu/ethereum/api/handlers/EngineJsonRpcParserHandler.java
Outdated
Show resolved
Hide resolved
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/handlers/JsonRpcParserHandler.java
Show resolved
Hide resolved
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…-requests Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
This reverts commit 36ad488. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
This reverts commit 7386e7a. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
This reverts commit 5eec0ba. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Similar to hyperledger#8807 Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
and increase default value from 5MB to 128MB This has a side effect of increasing the JsonRpcService limit. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…nabled Reuse maxRequestContentLength to keep aligned with HTTP body limit String length should be comparable with byte size assuming UTF-8 encoding Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
0387721 to
2287e02
Compare
…-requests Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
| public static final int DEFAULT_MAX_ACTIVE_CONNECTIONS = 80; | ||
| public static final int DEFAULT_MAX_BATCH_SIZE = 1024; | ||
| public static final long DEFAULT_MAX_REQUEST_CONTENT_LENGTH = 5 * 1024 * 1024; // 5MB | ||
| public static final long DEFAULT_MAX_REQUEST_CONTENT_LENGTH = 128 * 1024 * 1024; // 128MB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need two constants with the same name and same value in two different files ? Why not keep only one of these ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally, we should have one set of this config, but these duplicates are an existing issue, not to sort out for this PR IMO
ahamlat
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
6367ecc to
b3b79df
Compare
* docs: edited the badge that was displayed incorrectly (#8623) * Update README.md Signed-off-by: Olexandr88 <radole1203@gmail.com> * Update README.md Signed-off-by: Jason Frame <jasonwframe@gmail.com> --------- Signed-off-by: Olexandr88 <radole1203@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Jason Frame <jason.frame@consensys.net> * getBlobsV2 prohibits partial responses (#8967) * getBlobsV2 to not return partial responses Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Change precompile name in eth_config (#8986) * Change precompile name in eth_config The precompile names for bn254 were encoded as bn256, which is a widely used but incorrect name. The EIP specified alt_bn128, for 128 bits of security, but cryptographic analysis has shown the security to be less than 128 bits, so the name referring to the field size is preferred. Signed-off-by: Danno Ferrin <danno@numisight.com> * fixed test Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * remove separate qbft ATs gradle task and workflow (#8962) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * remove deprecated method Quantity.getValue (#8968) * update tests to use non-deprecated method * remove deprecated method * remove getValue methods that overrode the removed method Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Clean up production of the fastSyncFactory in DefaultSynchronizer (#8975) * Clean up production of the fastSyncFactory in DefaultSynchronizer Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix NPE caused by using a mock SynchronizerConfiguration Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix NPE caused by using a mock SynchronizerConfiguration Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * spotless Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Avoid using syncMode directly Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> --------- Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * remove deprecated PoAMetricsService and IbftQueryService (#8940) * remove deprecated interfaces * update plugin api hash Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * refactor: central control point for genesis used in tests (#8819) * made a constant Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Rename block hash processor (#8993) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Generate distribution dependencies catalog (#8987) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * feat: add world view to traceStartBlock hook (#8981) * Add world view to trace start block Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * change known hash Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix unit tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Do not account for system calls in the tracer (#8992) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Remove PoW ATs which were replaced by QBFT miner based ATs (#8949) * Delete PoW ATs which were replaced by QBFT miner based ATs * Change another AT to used QBFT genesis * update node config to fix couple of AT failures * Update transaction validator AT to use 4 validators in QBFT cluster * Update Besueventsplugintest * Revert an accidentally deleted AT and now modified to use QBFT Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> --------- Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Improve BENCHMARKING.md documentation (#9001) Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net> * Docker labels update (#9002) * fix docker build Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * update docker labels to the new oci standard Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> --------- Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * No RLP decoding of Bodies during the sync (fix #8242) (#8959) * By not decoding the bodies we use less memory and CPU, which allows us to increase the chain download parallelism. Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> * Enable decoding for large RPC requests (#8877) Use maxRequestContentLength for EngineJsonRpcService bodyLimit and increase default value from 5MB to 128MB. This has a side effect of increasing the JsonRpcService limit. Use maxRequestContentLength to globally set maxStringLength for Vertx ObjectMapper when engine API enabled. Reusing maxRequestContentLength to keep aligned with HTTP body limit. String length should be comparable with byte size assuming UTF-8 encoding. Also remove redundant serialization of params for optionalList Similar to #8807 Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add --attempt-cache-bust to evmtool benchmark subcommand (#8985) This means running all the test cases within one warmup or exec iteration in an attempt to invalidate CPU caches. Requires tracking exec timings in a map and calculating average at the end. Reduce warmup iteration to 20_000 and execution iterations to 1000, which appears to be as stable as 100_000. This helps reduces overall benchmark execution time (previously warmTime and execTime were prematurely reducing overall execution time). Use SequencedMap to indicate test case ordering matters. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * TransactionDecoder: handle empty bytes (#8997) * handle empty bytes * check for empty before getting tx type Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Add a message for disconnect message reasons (#9014) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Remove the possibility to create block before Byzantium hard-fork (#9005) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * chore: remove support for eth/67 (#9008) * remove eth/67 capability Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix naming Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * changelog Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix mock test Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Update CHANGELOG.md Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Improve info logs during block creation (#9020) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Chore: Prevent repeated cli boolean flags for being accepted (#8898) * Chore: Prevent repeated cli boolean flags for being accepted Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> * allow unmatched options Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> * Eliminate AbstractRLPInput abstraction to reduce technical debt (#8885) Merge AbstractRLPInput (607 lines) and BytesValueRLPInput (84 lines) into a single concrete class implementing RLPInput directly. This eliminates the unnecessary abstraction layer where the abstract class had only one concrete implementation. Changes: - Merged all AbstractRLPInput logic into BytesValueRLPInput - Made previously protected methods private - Removed AbstractRLPInput.java file - All public APIs remain unchanged (zero breaking changes) - All tests pass unchanged Benefits: - Eliminates 15-30 minutes investigation time per RLP modification - Reduces cognitive load when working with RLP functionality - Simplifies maintenance with single concrete implementation - No performance or functionality impact Fixes #8884 🤖 Generated with [Claude Code](https://claude.ai/code) Signed-off-by: jflo <justin+github@florentine.us> Co-authored-by: Claude <noreply@anthropic.com> * Add WorldStateService to the plugin API (#9024) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Change eth_config hashing and config members (#9015) * Change eth_config hashing and config members - Remove ConfigHash - move ForkID into config - invert key/value of precompiles Signed-off-by: Danno Ferrin <danno@numisight.com> * linting Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Era1 import prepipeline qol improvements (#8964) * Add --era1-import-prepipeline-enabled and --era1-data-path flags Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Implement ERA1 file import prepipeline for full sync Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Filter out block 0 as this is included by default Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix SynchronizerOptionsTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * spotless Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix BesuCommandTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add ability to load ERA1 files direct from HTTP source, fix bug in Era1Reader exposed by using HTTP source Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix Era1ReaderTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add javadoc to InputStreamFactory Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add default constructor to InputStreamFactory Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix Era1FileSourceTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Only produce era1 import prepipeline if sync mode is FULL Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Make era1 prepipeline concurrency configurable Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use Integer instead of int in SynchronizerOptions Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Update era1 import prepipeline concurrency to 1 Since the bottleneck for this pipeline is typically the import block step, there is no need for more concurrency than a single pipeline gives. Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> * Inline starting full sync chain downloader Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Simplify comment Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use constant for era1 block count Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use Era1FileReader as a launching point for more worker threads to achieve better parallelism Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> --------- Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> * Eliminate AbstractRLPOutput abstraction to reduce technical debt (#8890) - Consolidate AbstractRLPOutput logic into BytesValueRLPOutput - Remove unnecessary inheritance layer with single implementation - Maintain identical public API and functionality - Zero breaking changes to 150+ usage sites across codebase This follows the same pattern as recent AbstractRLPInput elimination, completing the removal of premature abstractions in the RLP module. Fixes #8889 🤖 Generated with [Claude Code](https://claude.ai/code) Signed-off-by: jflo <justin+github@florentine.us> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Olexandr88 <radole1203@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net> Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> Signed-off-by: jflo <justin+github@florentine.us> Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> Co-authored-by: Olexandr88 <radole1203@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Jason Frame <jason.frame@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Danno Ferrin <danno@numisight.com> Co-authored-by: Matilda-Clerke <matilda.clerke@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Bhanu Pulluri <59369753+pullurib@users.noreply.github.com> Co-authored-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net> Co-authored-by: Joshua Fernandes <joshua.fernandes@consensys.net> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Preeti <35308865+pr9t@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
* docs: edited the badge that was displayed incorrectly (#8623) * Update README.md Signed-off-by: Olexandr88 <radole1203@gmail.com> * Update README.md Signed-off-by: Jason Frame <jasonwframe@gmail.com> --------- Signed-off-by: Olexandr88 <radole1203@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Jason Frame <jason.frame@consensys.net> * getBlobsV2 prohibits partial responses (#8967) * getBlobsV2 to not return partial responses Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Change precompile name in eth_config (#8986) * Change precompile name in eth_config The precompile names for bn254 were encoded as bn256, which is a widely used but incorrect name. The EIP specified alt_bn128, for 128 bits of security, but cryptographic analysis has shown the security to be less than 128 bits, so the name referring to the field size is preferred. Signed-off-by: Danno Ferrin <danno@numisight.com> * fixed test Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * remove separate qbft ATs gradle task and workflow (#8962) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * remove deprecated method Quantity.getValue (#8968) * update tests to use non-deprecated method * remove deprecated method * remove getValue methods that overrode the removed method Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Clean up production of the fastSyncFactory in DefaultSynchronizer (#8975) * Clean up production of the fastSyncFactory in DefaultSynchronizer Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix NPE caused by using a mock SynchronizerConfiguration Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix NPE caused by using a mock SynchronizerConfiguration Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * spotless Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Avoid using syncMode directly Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> --------- Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * remove deprecated PoAMetricsService and IbftQueryService (#8940) * remove deprecated interfaces * update plugin api hash Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * refactor: central control point for genesis used in tests (#8819) * made a constant Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Rename block hash processor (#8993) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Generate distribution dependencies catalog (#8987) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * feat: add world view to traceStartBlock hook (#8981) * Add world view to trace start block Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * change known hash Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix unit tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Do not account for system calls in the tracer (#8992) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Remove PoW ATs which were replaced by QBFT miner based ATs (#8949) * Delete PoW ATs which were replaced by QBFT miner based ATs * Change another AT to used QBFT genesis * update node config to fix couple of AT failures * Update transaction validator AT to use 4 validators in QBFT cluster * Update Besueventsplugintest * Revert an accidentally deleted AT and now modified to use QBFT Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> --------- Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Improve BENCHMARKING.md documentation (#9001) Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net> * Docker labels update (#9002) * fix docker build Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * update docker labels to the new oci standard Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> --------- Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * No RLP decoding of Bodies during the sync (fix #8242) (#8959) * By not decoding the bodies we use less memory and CPU, which allows us to increase the chain download parallelism. Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> * Enable decoding for large RPC requests (#8877) Use maxRequestContentLength for EngineJsonRpcService bodyLimit and increase default value from 5MB to 128MB. This has a side effect of increasing the JsonRpcService limit. Use maxRequestContentLength to globally set maxStringLength for Vertx ObjectMapper when engine API enabled. Reusing maxRequestContentLength to keep aligned with HTTP body limit. String length should be comparable with byte size assuming UTF-8 encoding. Also remove redundant serialization of params for optionalList Similar to #8807 Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add --attempt-cache-bust to evmtool benchmark subcommand (#8985) This means running all the test cases within one warmup or exec iteration in an attempt to invalidate CPU caches. Requires tracking exec timings in a map and calculating average at the end. Reduce warmup iteration to 20_000 and execution iterations to 1000, which appears to be as stable as 100_000. This helps reduces overall benchmark execution time (previously warmTime and execTime were prematurely reducing overall execution time). Use SequencedMap to indicate test case ordering matters. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * TransactionDecoder: handle empty bytes (#8997) * handle empty bytes * check for empty before getting tx type Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Add a message for disconnect message reasons (#9014) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Remove the possibility to create block before Byzantium hard-fork (#9005) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * chore: remove support for eth/67 (#9008) * remove eth/67 capability Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix naming Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * changelog Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix mock test Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * fix tests Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Update CHANGELOG.md Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Improve info logs during block creation (#9020) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Chore: Prevent repeated cli boolean flags for being accepted (#8898) * Chore: Prevent repeated cli boolean flags for being accepted Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> * allow unmatched options Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> * Eliminate AbstractRLPInput abstraction to reduce technical debt (#8885) Merge AbstractRLPInput (607 lines) and BytesValueRLPInput (84 lines) into a single concrete class implementing RLPInput directly. This eliminates the unnecessary abstraction layer where the abstract class had only one concrete implementation. Changes: - Merged all AbstractRLPInput logic into BytesValueRLPInput - Made previously protected methods private - Removed AbstractRLPInput.java file - All public APIs remain unchanged (zero breaking changes) - All tests pass unchanged Benefits: - Eliminates 15-30 minutes investigation time per RLP modification - Reduces cognitive load when working with RLP functionality - Simplifies maintenance with single concrete implementation - No performance or functionality impact Fixes #8884 🤖 Generated with [Claude Code](https://claude.ai/code) Signed-off-by: jflo <justin+github@florentine.us> Co-authored-by: Claude <noreply@anthropic.com> * Add WorldStateService to the plugin API (#9024) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Change eth_config hashing and config members (#9015) * Change eth_config hashing and config members - Remove ConfigHash - move ForkID into config - invert key/value of precompiles Signed-off-by: Danno Ferrin <danno@numisight.com> * linting Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Era1 import prepipeline qol improvements (#8964) * Add --era1-import-prepipeline-enabled and --era1-data-path flags Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Implement ERA1 file import prepipeline for full sync Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Filter out block 0 as this is included by default Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix SynchronizerOptionsTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * spotless Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix BesuCommandTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add ability to load ERA1 files direct from HTTP source, fix bug in Era1Reader exposed by using HTTP source Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix Era1ReaderTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add javadoc to InputStreamFactory Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Add default constructor to InputStreamFactory Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Fix Era1FileSourceTest Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Only produce era1 import prepipeline if sync mode is FULL Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Make era1 prepipeline concurrency configurable Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use Integer instead of int in SynchronizerOptions Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Update era1 import prepipeline concurrency to 1 Since the bottleneck for this pipeline is typically the import block step, there is no need for more concurrency than a single pipeline gives. Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> * Inline starting full sync chain downloader Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Simplify comment Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use constant for era1 block count Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> * Use Era1FileReader as a launching point for more worker threads to achieve better parallelism Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> --------- Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> * Eliminate AbstractRLPOutput abstraction to reduce technical debt (#8890) - Consolidate AbstractRLPOutput logic into BytesValueRLPOutput - Remove unnecessary inheritance layer with single implementation - Maintain identical public API and functionality - Zero breaking changes to 150+ usage sites across codebase This follows the same pattern as recent AbstractRLPInput elimination, completing the removal of premature abstractions in the RLP module. Fixes #8889 🤖 Generated with [Claude Code](https://claude.ai/code) Signed-off-by: jflo <justin+github@florentine.us> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * [BESU-8973] Display configured target-gas-limit in startup logs (#8998) * [BESU-8973] Fix issue #8973 * Added `targetGasLimit` property and related setter to `ConfigurationOverviewBuilder` class * Added line to `build` function for `targetGasLimit` in `ConfigurationOverviewBuilder` class * Added set of `targetGasLimit` for `builder` object in `generateConfigurationOverview` of `BesuCommand` class * Set default value for `targetGasLimit` in `BesuCommand` class * Changed visibility of `normalizeGas` in `ConfigurationOverviewBuilder` class for usage in tests * Added `getDefaultGasLimitByChainId` method in `MergeCoordinator` * Added tests for `MergeCoordinator`, `ConfigurationOverviewBuilder` and `BesuCommandTest` * Tests for mainnet, testnet and dev with default and custom targetGasLimit * Replaced Sepolia with Hoodi for MergeCoordinatorTest Signed-off-by: Gabriele Del Monte <gabriele.delmonte@proton.me> * spotless Signed-off-by: Simon Dudley <simon.dudley@consensys.net> --------- Signed-off-by: Gabriele Del Monte <gabriele.delmonte@proton.me> Signed-off-by: Gabriele Del Monte <37625739+gabrieledm@users.noreply.github.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * [CHANGELOG] consolidate upcoming breaking changes (#9030) * remove unnecessary use of deprecated options * changelog Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * use dummy coinbase in simulation if not defined (#9031) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Use system call for beacon root (#8982) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Backward sync - check we have peers before starting (#9003) * check if ready before first step * add peer count > 0 check to isReady * added a test for zero peers Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Fix mixHash / prevRandao deserialization in BlockOverrides (#8983) * Fix mixHash / prevRandao deserialization in BlockOverrides Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Update plugin-api checksum Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Add deserialization test failing on main and passing on this branch Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Update ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/parameters/BlockOverridesParameterTest.java Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Apply formatter Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> --------- Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Feat: add gas usage metric on eth_call (#9019) * WIP: update EthCall.java to capture gas usage metric * update EthJsonRpcMethods.java and JsonRpcMethodsFactory.java constructors to include metricsSystem * Update EthCallTest.java to mock gas usage metric * Update transactionSimulator error label to internal_error * Update CHANGELOG.md Signed-off-by: Reuben George <reubengeorge101@gmail.com> --------- Signed-off-by: Reuben George <reubengeorge101@gmail.com> Signed-off-by: Reuben George <69674713+georgereuben@users.noreply.github.com> Co-authored-by: Karim Taam <karim.t2am@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Remove CLI options for JMH and adapt benchmarks (#9025) PR allows for changing JMH settings in benchmarks themselves instead of gradle plugin but fixes forks to 3 for all benchamrks. Also done some tuning of CLZ and Keccak benchmarks. * [CHARTER] make explicit the voting for emeritus process (#8937) * make explicit the voting for emeritus process Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * headings formatting Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> * Bump prioritized transactions for blobs (#9037) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Removed BlockExplorer and AbstractBlockScheduler due single implementation. (#9026) Signed-off-by: Marcos Serradilla Diez <marcos@io.builders> * sonarcloud failing due to deprecated actions/cache version (#9036) * sonarcloud failing due to deprecated actions/cache version Signed-off-by: jflo <justin+github@florentine.us> * new release just dropped Signed-off-by: jflo <justin+github@florentine.us> --------- Signed-off-by: jflo <justin+github@florentine.us> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Include authorization list in the transaction call object and use it when building transaction during simulation (#8984) * Include code delegation authorizations in the call parameter and use when building transaction during simulation Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Add authorizationList deserialization test Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Test that transaction type is correct if authorizations present in call param Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Add javadoc Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Extract TransactionCallParameterBuilder creation to a separate method for consistency Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> * Include codeDelegationAuthorizations when building Transaction from CallParameter Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> --------- Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Improve P256Verify precompile performance (#9035) - Upgrade besu-native to 1.4.0. - Use BoringSSL by default and fallback to SECP256R1 signature algorithm which has its own native library (as well as Java fallback). - Add --Xp256verify-native-enabled to disable BoringSSL. - P256VerifyBenchmark switches between BoringSSL and Java. - Fixup P256VerifyBenchmark testcase names so they don't override each other in the Map. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * update checkout action to 4.2.2 (#9052) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * EngineNewPayload - respond with error if params invalid (#8729) * added test for duplicate requests * most errors don't result in INVALID status * return invalid for invalid block * new error code * changelog entry Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> * Fix merge main Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Olexandr88 <radole1203@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Danno Ferrin <danno@numisight.com> Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net> Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Preeti <35308865+pr9t@users.noreply.github.com> Signed-off-by: jflo <justin+github@florentine.us> Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net> Signed-off-by: Gabriele Del Monte <gabriele.delmonte@proton.me> Signed-off-by: Gabriele Del Monte <37625739+gabrieledm@users.noreply.github.com> Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com> Signed-off-by: Reuben George <reubengeorge101@gmail.com> Signed-off-by: Reuben George <69674713+georgereuben@users.noreply.github.com> Signed-off-by: Marcos Serradilla Diez <marcos@io.builders> Co-authored-by: Olexandr88 <radole1203@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Jason Frame <jason.frame@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Danno Ferrin <danno@numisight.com> Co-authored-by: Matilda-Clerke <matilda.clerke@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Bhanu Pulluri <59369753+pullurib@users.noreply.github.com> Co-authored-by: Bhanu Pulluri <bhanu.pulluri@kaleido.io> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net> Co-authored-by: Joshua Fernandes <joshua.fernandes@consensys.net> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Preeti <35308865+pr9t@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Gabriele Del Monte <37625739+gabrieledm@users.noreply.github.com> Co-authored-by: Miroslav Kovar <miroslavkovar@protonmail.com> Co-authored-by: Reuben George <69674713+georgereuben@users.noreply.github.com> Co-authored-by: Karim Taam <karim.t2am@gmail.com> Co-authored-by: Luis Pinto <luis.pinto@consensys.net> Co-authored-by: Marcos Serradilla Diez <marcos@io.builders> Co-authored-by: garyschulte <garyschulte@gmail.com>
Use maxRequestContentLength for EngineJsonRpcService bodyLimit and increase default value from 5MB to 128MB. This has a side effect of increasing the JsonRpcService limit. Use maxRequestContentLength to globally set maxStringLength for Vertx ObjectMapper when engine API enabled. Reusing maxRequestContentLength to keep aligned with HTTP body limit. String length should be comparable with byte size assuming UTF-8 encoding. Also remove redundant serialization of params for optionalList Similar to hyperledger#8807 Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Use maxRequestContentLength for EngineJsonRpcService bodyLimit and increase default value from 5MB to 128MB. This has a side effect of increasing the JsonRpcService limit. Use maxRequestContentLength to globally set maxStringLength for Vertx ObjectMapper when engine API enabled. Reusing maxRequestContentLength to keep aligned with HTTP body limit. String length should be comparable with byte size assuming UTF-8 encoding. Also remove redundant serialization of params for optionalList Similar to hyperledger#8807 Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: jflo <justin+github@florentine.us>
This is to support large payloads (in engine_newPayload) at least up to 200MGas block sizes (and a bit more). Limit may need revisiting at larger block sizes.
Max out StreamReadConstraints.maxStringLength for jackson decoding of request and parametersCreate Engine API specific EngineJsonRpcParserHandler and EngineJsonRpcParameter containing modified ObjectMapper.Go back to globally configured but using a 128MB equivalent limit instead of MAX_VALUE. This aligns with the bodyLimit we were already setting.
A version of this was tested on performance branch via #8791 and #8796
but this PR restricts the changes to Engine API so shouldn't open up external DOS vectors since Engine API is assumed to be restricted to internal networks.Why the large limit?
Using the large transaction full of zeros test cases from https://github.com/NethermindEth/gas-benchmarks/blob/main/tests-vm/TxDataZero30M_30M.txt :
128MB was already used as the engine API HTTP body limit, so lowering this is a potential breaking change.
128MB is roughly equivalent to 128M chars with UTF-8 encoding.
128MB will support 200Mgas transactions which are used in the performance benchmarking suite.
We should maybe increase this limit even higher if we want to support a 300Mgas full of zeros.
Testing
No visible impact on mainnet sync or operations
This PR is able to parse this large newPayload: https://github.com/NethermindEth/gas-benchmarks/blob/main/tests-vm/TxDataZero200M_200M.txt
Also wrote a custom test to cache this large payload and retrieve with getPayload.
Tested with ethereum/go-ethereum@master/cmd/workload and behaves the same as control.