Supporting writing schema metadata when writing Parquet in parallel#13866

Merged

alamb merged 16 commits intoapache:mainfrom

influxdata:11770/parquet-sink-metadata

Jan 1, 2025

Commits on Dec 20, 2024

refactor: make ParquetSink tests a bit more readable
wiedld
committed

Commits on Dec 21, 2024

chore(11770): add new ParquetOptions.skip_arrow_metadata
wiedld
committed
test(11770): demonstrate that the single threaded ParquetSink is already writing the arrow schema in the kv_meta, and allow disablement
wiedld
committed
refactor(11770): replace with new method, since the kv_metadata is inherent to TableParquetOptions and therefore we should explicitly make the API apparant that you have to include the arrow schem…
wiedld
committed
fix(11770): fix parallel ParquetSink to encode arrow schema into the file metadata, based on the ParquetOptions
wiedld
committed
refactor(11770): provide deprecation warning for TryFrom
wiedld
committed
test(11770): update tests with new default to include arrow schema
wiedld
committed

Commits on Dec 26, 2024

Commits on Dec 27, 2024

Commits on Dec 28, 2024

chore: add reference to arrow-rs upstream PR
wiedld
committed