ARROW-6519: [Java] Use IPC continuation prefix as part of 8-byte EOS #5345
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BryanCutler
commented
Sep 10, 2019
| out.writeIntLittleEndian(0); | ||
| } else { | ||
| out.writeLongLittleEndian(0); | ||
| public static void writeEndOfStream(WriteChannel out, IpcOption option) throws IOException { |
Member
Author
There was a problem hiding this comment.
I'd prefer to keep this static because it is being used here to hide the EOS identifier https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala#L67
Member
Author
BryanCutler
changed the title
BryanCutler
changed the title
tianchen92
reviewed
Sep 11, 2019
| out.writeLongLittleEndian(0); | ||
| public static void writeEndOfStream(WriteChannel out, IpcOption option) throws IOException { | ||
| if (!option.write_legacy_ipc_format) { | ||
| out.writeIntLittleEndian(MessageSerializer.IPC_CONTINUATION_TOKEN); |
Contributor
There was a problem hiding this comment.
Also apply this change to ArrowFileWriter#endInternal and probably remove MessageSerializer#writeLongLitterEndian since it’s no use anymore?
Member
Author
There was a problem hiding this comment.
good catch, I forgot that also writes EOS. done.
tianchen92
approved these changes
Sep 11, 2019
Contributor
tianchen92
left a comment
tianchen92
left a comment
There was a problem hiding this comment.
LGTM, thanks @BryanCutler
Member
|
Merging this. https://github.com/apache/arrow/tree/ARROW-6313-flatbuffer-alignment has a rebase conflict on apache/master, I'm going to try to fix |
wesm
force-pushed
the
ARROW-6313-flatbuffer-alignment
branch
from
4f9b887 to
0352456
Compare
Write IPC continuation token to file format EOS
wesm
force-pushed
the
java-ipc-cont-for-EOS-ARROW-6519
branch
from
edce1cd to
8fb38a0
Compare
wesm
pushed a commit
that referenced
this pull request
Sep 11, 2019
This changes the 8-byte EOS for non-legacy stream format to use {0xFFFFFFFF, 0x00000000} instead of all zeros. When using all zeros, the reader will not know to read the last 4-bytes, but with the 4-byte continuation token, all bytes written to a channel can be read.
Closes #5345 from BryanCutler/java-ipc-cont-for-EOS-ARROW-6519 and squashes the following commits:
8fb38a0 <Bryan Cutler> Use IPC continuation token to write 8-byte EOS
Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Member
Author
|
Thanks @wesm and @tianchen92 ! |
wesm
pushed a commit
that referenced
this pull request
Sep 13, 2019
This changes the 8-byte EOS for non-legacy stream format to use {0xFFFFFFFF, 0x00000000} instead of all zeros. When using all zeros, the reader will not know to read the last 4-bytes, but with the 4-byte continuation token, all bytes written to a channel can be read.
Closes #5345 from BryanCutler/java-ipc-cont-for-EOS-ARROW-6519 and squashes the following commits:
8fb38a0 <Bryan Cutler> Use IPC continuation token to write 8-byte EOS
Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
pprudhvi
pushed a commit
to pprudhvi/arrow
that referenced
this pull request
Sep 16, 2019
This changes the 8-byte EOS for non-legacy stream format to use {0xFFFFFFFF, 0x00000000} instead of all zeros. When using all zeros, the reader will not know to read the last 4-bytes, but with the 4-byte continuation token, all bytes written to a channel can be read.
Closes apache#5345 from BryanCutler/java-ipc-cont-for-EOS-ARROW-6519 and squashes the following commits:
8fb38a0 <Bryan Cutler> Use IPC continuation token to write 8-byte EOS
Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
pribor
pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
This changes the 8-byte EOS for non-legacy stream format to use {0xFFFFFFFF, 0x00000000} instead of all zeros. When using all zeros, the reader will not know to read the last 4-bytes, but with the 4-byte continuation token, all bytes written to a channel can be read.
Closes apache#5345 from BryanCutler/java-ipc-cont-for-EOS-ARROW-6519 and squashes the following commits:
8fb38a0 <Bryan Cutler> Use IPC continuation token to write 8-byte EOS
Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes the 8-byte EOS for non-legacy stream format to use {0xFFFFFFFF, 0x00000000} instead of all zeros. When using all zeros, the reader will not know to read the last 4-bytes, but with the 4-byte continuation token, all bytes written to a channel can be read.