Skip to content

Conversation

@siladu
Copy link
Contributor

@siladu siladu commented May 22, 2025

PR description

besu --Xhistory-expiry-prune
will automatically set the following rocksdb GC options based on the testing conducted in #8599:

--Xplugin-rocksdb-blockchain-blob-garbage-collection-enabled
--Xplugin-rocksdb-blob-garbage-collection-age-cutoff=0.5
--Xplugin-rocksdb-blob-garbage-collection-force-threshold=0.1

The config overview displays:

# History expiry prune enabled                                                                     #

and the full settings are shown in an INFO log:
2025-05-23 09:39:40.179+10:00 | main | INFO | RocksDBKeyValueStorageFactory | History expiry prune is enabled so setting --Xplugin-rocksdb-blockchain-blob-garbage-collection-enabled; --Xplugin-rocksdb-blob-garbage-collection-age-cutoff=0.5; --Xplugin-rocksdb-blob-garbage-collection-force-threshold=0.1

It is still possible to set these options separately.
It is also possible to set these options along with --Xhistory-expiry-prune as a way to override the configured settings (though this isn't recommended).

For example, if I do --Xhistory-expiry-prune --Xplugin-rocksdb-blob-garbage-collection-age-cutoff=0.9 to make the age cutoff consider all files for GC, then the user will see this displayed:

# History expiry prune enabled                                                                     #
# Experimental BlobDB GC age cutoff: 0.9;                                                          #

2025-05-23 09:44:45.810+10:00 | main | INFO | RocksDBKeyValueStorageFactory | History expiry prune is enabled so setting --Xplugin-rocksdb-blockchain-blob-garbage-collection-enabled; --Xplugin-rocksdb-blob-garbage-collection-age-cutoff=0.9; --Xplugin-rocksdb-blob-garbage-collection-force-threshold=0.1

Note, --Xplugin-rocksdb-blockchain-blob-garbage-collection-enabled is always forced to true when --Xhistory-expiry-prune is set because the BLOCKCHAIN column family must have GC enabled in order to reclaim space.

As per the GC flags PR #8599:

  • age-cutoff and force-threshold setting will apply to all column families with BlobDB enabled (currently just TRIE_LOG_STORAGE) but the impact should be minimal since if you are enabling history expiry, you likely also only retain recent Trie Logs.
  • we will only display the settings in the configuration overview if the user has explicitly set the GC flags, we don't display them when the --Xhistory-expiry-prune flag sets them. The INFO logs will always show the full settings.

Fixed Issue(s)

Fixes #8654

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • spotless: ./gradlew spotlessApply
  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests

…ksDB GC settings

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu siladu added the history reduce disk reqs thru history mgmt label May 22, 2025
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu siladu marked this pull request as ready for review May 23, 2025 00:00
@siladu siladu moved this to In progress in History Expiry May 23, 2025
@siladu siladu moved this from In progress to In review in History Expiry May 23, 2025
@siladu siladu self-assigned this May 23, 2025
@siladu siladu enabled auto-merge (squash) May 23, 2025 03:20
@siladu siladu merged commit 6de2bbf into hyperledger:main May 23, 2025
48 checks passed
@github-project-automation github-project-automation bot moved this from In review to Done in History Expiry May 23, 2025
@siladu siladu deleted the history-expiry-prune-flag branch May 23, 2025 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

history reduce disk reqs thru history mgmt

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Add history-expiry-prune convenience flag to wrap prune GC settings

2 participants