Skip to content

ARROW-998: [Format] Clarify that the IPC file footer contains an additional copy of the schema #694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wesm wants to merge 1 commit into from
Closed

ARROW-998: [Format] Clarify that the IPC file footer contains an additional copy of the schema #694

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions format/IPC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,11 @@ as an `int32` or simply closing the stream interface.
We define a "file format" supporting random access in a very similar format to
the streaming format. The file starts and ends with a magic string `ARROW1`
(plus padding). What follows in the file is identical to the stream format. At
the end of the file, we write a *footer* including offsets and sizes for each
of the data blocks in the file, so that random access is possible. See
[format/File.fbs][1] for the precise details of the file footer.
the end of the file, we write a *footer* containing a redundant copy of the
schema (which is a part of the streaming format) plus memory offsets and sizes
for each of the data blocks in the file. This enables random access any record
batch in the file. See [format/File.fbs][1] for the precise details of the file
footer.

Schematically we have:

Expand Down