Adding Iceberg Support#29569

Closed

byronellis wants to merge 15 commits intoapache:masterfrom

byronellis:iceberg-support

Commits on Nov 17, 2023

Iceberg test harness implementation
Byron Ellis
committed

Commits on Nov 21, 2023

WIP
Byron Ellis
committed

Commits on Nov 27, 2023

Building out scan support for schemas.
byronellis
committed

Commits on Nov 28, 2023

Merge branch 'apache:master' into iceberg-support
byronellis
authored
Add schema return
Byron Ellis
committed
Merge remote-tracking branch 'origin/iceberg-support' into iceberg-support
Byron Ellis
committed

Commits on Nov 30, 2023

Working extremely basic Parquet read
Byron Ellis
committed

Commits on Dec 19, 2023

WIP converting to BoundedSource. Added eventual Catalog interface (might split out)
Byron Ellis
committed

Commits on Dec 20, 2023

Added all configuration parameters and converted DoFns to a real BoundedScan. Swapped test to use that.
byronellis
committed

Commits on Feb 22, 2024

Added very basic sink. Need to do metadata updates externally
Byron Ellis
committed

Commits on Feb 24, 2024

No way to get FileBasedSink to work with catalog updates, copy pasta it into a new implementation instead.
Byron Ellis
committed

Commits on Mar 20, 2024

Refactor the sink into something that looks basically like BigQueryIO's BatchLoad implementation since this is a pretty close analog. Has the beginnings of dynamic destination support, though doesn…
Byron Ellis
committed

Commits on Mar 21, 2024