Skip to content

Update encoding.md for MiniBlockChunk minichunk size changes #5240

@niyue

Description

@niyue

PR #4959 expanded MiniBlockChunk minichunk size from u16 to u32, introducing new metadata and protobuf properties. The documentation at docs/src/format/file/encoding.md should be updated to reflect these changes:

  • New protobuf property: has_large_chunk for MiniBlockLayout.
  • New metadata key: lance-encoding:minichunk-size for specifying mini chunk size (default 4KB, can be larger, e.g., 64KB).
  • buffer_sizes now stored as u32 in memory:
    • Lance format v2.1: still stored as u16 on disk, converted to u32 in memory.
    • Lance format v2.2: default mini chunk size remains 4KB, but users can override it (up to 64KB or larger) via the new metadata key.

The docs should explain these changes so users and developers understand the updated MiniBlockChunk behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions