Skip to content

Static Files#6444

Merged
joshieDo merged 71 commits intobreaking-changesfrom
feat/static-files
Feb 27, 2024
Merged

Static Files#6444
joshieDo merged 71 commits intobreaking-changesfrom
feat/static-files

Conversation

@shekhirin
Copy link
Copy Markdown
Member

@shekhirin shekhirin commented Feb 6, 2024

Static files

Static files project aims to offer a place to store immutable/finalized data from a full fledged database such as mdbx into a mostly read-only store(1). Since it's a read-only store it can be accessed concurrently without 0 worries of data consistency and data can be compacted with no freelist and/or reserved unused space.

  • 0-worry concurrent access on historical data
  • Fast querying (random data points and ranges) (offset based indexes)
  • Append: happens during Pipeline on historical sync, or as a background job during live sync when copying finalized data from mdbx.
  • Truncate: happens during Pipeline unwind.
  • Divided by 500k range of blocks. (1): The latest static file might be truncated during a pipeline unwind.
  • Historical static file data can potentially be saved separately on a HDD or network drive serving multiple clients.

Sepolia stats

reth db stats at block #5127377

`main` (343.1GiB)

image

`feat/static-files` (311.5 GiB)

image
image

Related PRs:

https://github.com/paradigmxyz/reth/issues?q=+label%3AA-snapshots+

joshieDo and others added 22 commits January 18, 2024 18:18
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
Co-authored-by: Oliver Nordbjerg <hi@notbjerg.me>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
#6103)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Bjerg <onbjerg@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
@shekhirin shekhirin force-pushed the feat/static-files branch 2 times, most recently from 2f01b65 to b68288c Compare February 7, 2024 18:43
@shekhirin shekhirin force-pushed the feat/static-files branch 2 times, most recently from 97d77d1 to f8d4d85 Compare February 8, 2024 16:49
@joshieDo joshieDo marked this pull request as ready for review February 27, 2024 20:53
@joshieDo joshieDo merged commit 41c5f1a into breaking-changes Feb 27, 2024
@joshieDo joshieDo deleted the feat/static-files branch February 27, 2024 20:53
@shekhirin shekhirin changed the title WIP: static files Static Files Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-db Related to the database A-static-files Related to static files C-enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants