Skip to content

Add test summary reporting to block-test command #9246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
siladu merged 5 commits into from
Oct 15, 2025
Merged

Add test summary reporting to block-test command #9246

siladu merged 5 commits into from
Oct 15, 2025
+84 −15

Conversation

@bshastry
Copy link
Contributor

@bshastry bshastry commented Sep 30, 2025

PR description

This PR adds test summary reporting to the block-test command in evmtool. When processing large test suites (e.g., 1M block tests in a single JSON file), the command now tracks and displays a comprehensive summary at the end of execution, including:

  • Total number of tests executed
  • Count of passed tests
  • Count of failed tests
  • Detailed list of failed test names with their failure reasons

This enhancement significantly improves usability when running large blockchain test suites by providing a clear overview of test outcomes without requiring manual parsing of verbose output logs.

Example Output

================================================================================
TEST SUMMARY
================================================================================
Total tests:  1000
Passed:       995
Failed:       5

Failed tests:
  - invalidBlockTest: Block 5 (0xabc...) Failed to import
  - chainMismatchTest: Chain header mismatch, have 0x123... want 0x456...
================================================================================

Fixed Issue(s)

N/A - Enhancement request

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • spotless: ./gradlew spotlessApply
  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests
  • hive tests: Engine or other RPCs modified?

@bshastry bshastry force-pushed the feature/block-test-summary branch from 4383107 to 4796a1c Compare September 30, 2025 10:13
@bshastry bshastry force-pushed the feature/block-test-summary branch 3 times, most recently from a680c29 to 2cf63aa Compare October 8, 2025 09:10
@bshastry bshastry requested a review from usmansaleem October 8, 2025 09:11
@usmansaleem usmansaleem self-assigned this Oct 8, 2025
bshastry and others added 2 commits October 9, 2025 09:34
@bshastry @claude
Add test summary reporting to block-test command
0cf4847 
The block-test command now tracks and displays a summary of test
execution results, including total tests, passed/failed counts, and
a list of failed tests with their failure reasons. This improves
usability when running large test suites by providing a clear
overview of test outcomes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
@bshastry @claude
Refactor block-test summary output for better maintainability
1c51a4f 
Address code review feedback by:
- Extract separator string into SEPARATOR constant to reduce duplication
- Reuse failureReason variable in output statements instead of duplicating
  formatting logic

This improves code maintainability by consolidating the failure message
formatting in a single location.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
@bshastry bshastry force-pushed the feature/block-test-summary branch from 2cf63aa to 4877f10 Compare October 9, 2025 07:34
@bshastry @claude
Address review feedback: simplify TestResults implementation
f3b6224 
- Remove unused testName parameter from recordPass() method
- Remove totalTests field and calculate it dynamically in printSummary()
  from passed + failed counts
- Simplify SEPARATOR constant by removing embedded newline and use
  explicit out.println() for consistent formatting
- Add hasTests() helper method to check if any tests were executed

This reduces redundant state tracking and improves code clarity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
@bshastry bshastry force-pushed the feature/block-test-summary branch from 4877f10 to f3b6224 Compare October 9, 2025 07:34
@bshastry
Copy link
Contributor Author

bshastry commented Oct 9, 2025

Thank you for your feedback, sorry for the incomplete changes in my previous push, hope the latest version is better :)

@bshastry bshastry requested a review from usmansaleem October 9, 2025 07:35
usmansaleem
usmansaleem approved these changes Oct 15, 2025
View reviewed changes
Copy link
Member

@usmansaleem usmansaleem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@macfarla macfarla assigned macfarla and unassigned usmansaleem Oct 15, 2025
@siladu siladu self-requested a review October 15, 2025 01:01
siladu
siladu approved these changes Oct 15, 2025
View reviewed changes
Copy link
Contributor

@siladu siladu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice addition!

Wondering if it would be an improvement to include an average throughput in MGas/s in the summary? That can be a separate PR though.

macfarla
macfarla approved these changes Oct 15, 2025
View reviewed changes
Copy link
Contributor

@macfarla macfarla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@siladu siladu enabled auto-merge (squash) October 15, 2025 01:42
@siladu siladu merged commit 72ddaaf into hyperledger:main Oct 15, 2025
46 checks passed
@ahamlat
Copy link
Contributor

ahamlat commented Oct 15, 2025 •
edited
Loading

LGTM, nice addition!

Wondering if it would be an improvement to include an average throughput in MGas/s in the summary? That can be a separate PR though.

The mgas/s is already displayed with block-test for each imported block. It was added here : #9147.
But maybe your comment is more on having an average across all the executions. In this case, it would better to have min, 50th percentile and max throughtput in the summary.

bshastry added a commit to bshastry/besu that referenced this pull request Oct 15, 2025
@bshastry @claude
Add transaction tracing support to block test command
d35158c 
Implements tracing functionality for the block-test subcommand in evmtool,
allowing developers to debug block test execution with detailed transaction
traces including opcodes, gas usage, stack, and memory state.

New features:
- Added -t/--trace-transactions flag to enable tracing during block test execution
- Added --trace-memory, --trace-stack, --trace-returndata, --trace-storage options
- Added --trace-output option to specify output file (default: stderr)
- Integrated with StandardJsonTracer infrastructure
- Traces are output in JSON format compatible with go-ethereum

Implementation:
The processBlockWithTracing() method processes transactions with full tracing
while maintaining correct transaction status handling. Transactions are
categorized as INVALID, FAILED, or SUCCESSFUL per Ethereum protocol:
- INVALID transactions (wrong nonce, insufficient balance) reject the block
- FAILED transactions (reverted execution) are included with receipts
- SUCCESSFUL transactions are included with receipts

This distinction ensures blocks with reverted transactions are correctly
accepted, matching behavior of the standard block import path and reference
implementations like go-ethereum.

Conflict resolution:
This commit was rebased onto main which added test summary reporting (hyperledger#9246).
Both features are now merged:
- Test summary reporting tracks pass/fail counts across all tests
- Transaction tracing provides detailed execution traces when enabled
- Both features work independently and complement each other

Usage:
  evmtool block-test -t --trace-stack --trace-memory <test-file.json>
  evmtool block-test -t --trace-output=trace.jsonl <test-file.json>

Tested with Ethereum reference test suite including complex scenarios with
failed transactions. Trace output format matches go-ethereum for compatibility
with existing tooling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
bshastry added a commit to bshastry/besu that referenced this pull request Oct 17, 2025
@bshastry @claude
Add transaction tracing support to block test command
bd27f72 
Implements tracing functionality for the block-test subcommand in evmtool,
allowing developers to debug block test execution with detailed transaction
traces including opcodes, gas usage, stack, and memory state.

New features:
- Added -t/--trace-transactions flag to enable tracing during block test execution
- Added --trace-memory, --trace-stack, --trace-returndata, --trace-storage options
- Added --trace-output option to specify output file (default: stderr)
- Integrated with StandardJsonTracer infrastructure
- Traces are output in JSON format compatible with go-ethereum

Implementation:
The processBlockWithTracing() method processes transactions with full tracing
while maintaining correct transaction status handling. Transactions are
categorized as INVALID, FAILED, or SUCCESSFUL per Ethereum protocol:
- INVALID transactions (wrong nonce, insufficient balance) reject the block
- FAILED transactions (reverted execution) are included with receipts
- SUCCESSFUL transactions are included with receipts

This distinction ensures blocks with reverted transactions are correctly
accepted, matching behavior of the standard block import path and reference
implementations like go-ethereum.

Conflict resolution:
This commit was rebased onto main which added test summary reporting (hyperledger#9246).
Both features are now merged:
- Test summary reporting tracks pass/fail counts across all tests
- Transaction tracing provides detailed execution traces when enabled
- Both features work independently and complement each other

Usage:
  evmtool block-test -t --trace-stack --trace-memory <test-file.json>
  evmtool block-test -t --trace-output=trace.jsonl <test-file.json>

Tested with Ethereum reference test suite including complex scenarios with
failed transactions. Trace output format matches go-ethereum for compatibility
with existing tooling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
pinges pushed a commit to pinges/besu that referenced this pull request Dec 15, 2025
@bshastry @claude
@usmansaleem @siladu @pinges
Add test summary reporting to block-test command (hyperledger#9246)
6b61de1 
* Add test summary reporting to block-test command

The block-test command now tracks and displays a summary of test
execution results, including total tests, passed/failed counts, and
a list of failed tests with their failure reasons. This improves
usability when running large test suites by providing a clear
overview of test outcomes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Refactor block-test summary output for better maintainability

Address code review feedback by:
- Extract separator string into SEPARATOR constant to reduce duplication
- Reuse failureReason variable in output statements instead of duplicating
  formatting logic

This improves code maintainability by consolidating the failure message
formatting in a single location.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Address review feedback: simplify TestResults implementation

- Remove unused testName parameter from recordPass() method
- Remove totalTests field and calculate it dynamically in printSummary()
  from passed + failed counts
- Simplify SEPARATOR constant by removing embedded newline and use
  explicit out.println() for consistent formatting
- Add hasTests() helper method to check if any tests were executed

This reduces redundant state tracking and improves code clarity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

---------

Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@usmansaleem usmansaleem usmansaleem approved these changes

@macfarla macfarla macfarla approved these changes

@siladu siladu siladu approved these changes

Assignees

@macfarla macfarla

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bshastry @ahamlat @usmansaleem @macfarla @siladu