ARROW-5182: [Rust] Arrow IPC file writer #6107
Conversation
rust/arrow/src/ipc/writer.rs
Outdated
| let mut fbb = FlatBufferBuilder::new(); | ||
| let dictionaries = fbb.create_vector(&self.dictionary_blocks); | ||
| let record_batches = fbb.create_vector(&self.record_blocks); | ||
| // TODO: this is duplicated as we otherwise mutably borrow twice |
There was a problem hiding this comment.
I need help here, I couldn't find a way of reusing the functions in crate::ipc::convert
There was a problem hiding this comment.
I pulled this change down and played with it; I'm pretty excited to get the IPC writer for Rust :). I've run into similar challenges before with flatbuffers, and sometimes there's no good way of working around it.
Would it be worth converting schema_to_fb_offset to a macro maybe?
There was a problem hiding this comment.
I can try the macro approach if I can't find another workaround, I could do this when working on the StreamWriter given that it'll share most code with the FileWriter.
|
Hi @nevi-me I am back at work tomorrow and will start reviewing this. |
|
Hi! Is there anything I could be able to do to help with this merge? |
|
Hey @maxburke, the next release is a few days away, so I doubt this will make the cut. If you need to use the IPC writer for now, you can work from my branch in the short term. I can keep this PR updated with any other Rust changes in master. Do you also need a |
rust/arrow/src/ipc/writer.rs
Outdated
| /// A reference to the schema, used in validating record batches | ||
| schema: Schema, | ||
| /// The number of bytes written for the header (up to schema) | ||
| header_bytes: usize, |
There was a problem hiding this comment.
ended up not using this, will remove it as it duplicates block_offsets
|
The StreamWriter would be great to have but I've mostly worked around it for now with the FileWriter :) I'm actually currently working from your branch already. Really appreciate your help and progress on this! |
|
@andygrove I'm just waiting for a review |
|
@nevi-me After merging to master, I'm seeing a build failure in master: |
|
It's also happening on other PRs, I haven't had a chance to look at it. I'll look into it tomorrow morning |
The stream writer is applied on top of ARROW-5182 #6107 (file writer) and should be merged after. Closes #6281 from nevi-me/ARROW-7475 and squashes the following commits: 11ed168 <Neville Dipale> fix IPC test file location 0f90b3e <Neville Dipale> ARROW-7475: Arrow IPC Stream writer cb82160 <Neville Dipale> remove redundant header_bytes Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Andy Grove <andygrove73@gmail.com>
3 participants
This is committed on top of the PR for the stream reader, and should be merged after it.