[Format] Tracking for ensuring flatbuffer serialized values are aligned in stream/files. #22693
Description
Overall tracking bug for implementation for IPC/File format proposed by: https://github.com/apache/arrow/pull/4951/files
Implementations must support backwards compatibility with the old format (and ideally do memcopies when required to avoid undefined behavior). Having a backwards compatible write mode is optional
Reporter: Micah Kornfield / @emkornfield
Assignee: Wes McKinney / @wesm
Subtasks:
- [C++] Implement changes to ensure flatbuffer alignment.
- [Java] Make change to ensure flatbuffer reads are aligned
- [Go] Make change to ensure flatbuffer reads are aligned
- [JS] Implement changes to ensure flatbuffer alignment
- [Integration] Update integration test to use generated binaries to ensure backwards compatibility
- [Python] Provide mechanism for python to write out old format
- [Java] Use IPC continuation token to mark EOS
- [R] Provide mechanism to write out old format
Related issues:
- [Format][C++] Use two-part EOS and amend Format documentation (is a parent of)
- [Go] Update Go IPC writer to use two-part EOS per mailing list discussion (is a parent of)
- [Rust] Implement changes to ensure flatbuffer alignment (incorporates)
PRs and other links:
Note: This issue was originally created as ARROW-6313. Please see the migration documentation for further details.