[Rust] [Parquet] Implement parquet writer #24603
Description
This is the parent story. See subtasks for more information.
Notes from @wesm :
A couple of initial things to keep in mind
-
Writes of both Nullable (OPTIONAL) and non-nullable (REQUIRED) fields
-
You can optimize the special case where a nullable field's data has no nulls
-
A good amount of code is required to handle converting from the Arrow physical form of various logical types to the Parquet equivalent one, see https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_writer.cc for details
-
It would be worth thinking up front about how dictionary-encoded data is handled both on the Arrow write and Arrow read paths. In parquet-cpp we initially discarded Arrow DictionaryArrays on write (casting e.g. Dictionary to dense String), and through real world need I was forced to revisit this (quite painfully) to enable Arrow dictionaries to survive roundtrips to Parquet format, and also achieve better performance and memory use in both reads and writes. You can certainly do a dictionary-to-dense conversion like we did, but you may someday find yourselves doing the same painful refactor that I did to make dictionary write and read not only more efficient but also dictionary order preserving.
Notes from
[~sunchao]:I roughly skimmed through the C++ implementation and think on the high level we need to do the following:
- implement a method similar to
WriteArrowin column_writer.cc. We can further break this up into smaller pieces such as: dictionary/non-dictionary, primitive types, booleans, timestamps, dates, so on and so forth. - implement an arrow writer in the parquet crate here. This needs to offer similar APIs as writer.h.
Reporter: Andy Grove / @andygrove
Assignee: Neville Dipale / @nevi-me
Subtasks:
- [Rust] [Parquet] Implement minimal Arrow Parquet writer as starting point for full writer
- [Rust] [Parquet] Implement function to convert Arrow schema to Parquet schema
- [Rust] [Parquet] Serialize arrow schema into metadata when writing parquet
- [Rust] [Parquet] Add support for writing sliced arrays
- [Rust] [Parquet] Add support for writing temporal types
- [Rust] [Parquet] Add support for writing dictionary types
- [Rust] [Parquet] Compute nested definition and repetition for structs
- [Rust] [Parquet] Update for IPC changes
- [Rust] [Parquet] Extend arrow schema conversion to projected fields
- [Rust] [Parquet] Add roundtrip tests for single column batches
- [Rust] [Parquet] Fix null bitmap comparisons in roundtrip tests
- [Rust] [Parquet] Support reading and writing Arrow NullArray
- [Rust] [Parquet] Write nested types (struct, list)
- [Rust] [Parquet] Add support for writing boolean type
- [Rust] Compute nested definition and repetition for list arrays
- [Rust] [Parquet] Write fixed size binary arrays
PRs and other links:
Note: This issue was originally created as ARROW-8421. Please see the migration documentation for further details.