Skip to content

Implement per-page compression, page index#61

Merged
benbjohnson merged 3 commits intomainfrom
page-index
Aug 5, 2025
Merged

Implement per-page compression, page index#61
benbjohnson merged 3 commits intomainfrom
page-index

Conversation

@benbjohnson
Copy link
Collaborator

This pull request makes several backward incompatible changes to the LTX file format and bumps the Version number to 2.

  • Always compresses data (and there for removed the CompressLZ4 flag)
  • Compresses each page individually rather than compressing the entire page block.
  • Writes an index to the trailer of the file so that clients can maintain an index of page locations without downloading the entire file.

These changes will allow us to run a read replica VFS that pulls page data directly from S3 rather than restoring a full snapshot first.

@benbjohnson benbjohnson merged commit 9717266 into main Aug 5, 2025
2 checks passed
@benbjohnson benbjohnson deleted the page-index branch August 5, 2025 20:24
corylanou added a commit that referenced this pull request Aug 5, 2025
This test was already merged in PR #61 and should not be included
in this CI/linting PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant