feat!(datafusion): enable parallel file scanning with eager task bucketing#2298

Open

toutane wants to merge 16 commits intoapache:mainfrom

toutane:draft/partitioned-file-scanning-contribution

Commits on Apr 29, 2026

feat(datafusion): add IcebergPartitionedTableProvider and IcebergPartitionedScan for parallel file scanning
toutane
committed
docs(datafusion): update comment in IcebergPartitionedScan
toutane
committed
Update crates/integrations/datafusion/src/table/mod.rs

toutane
and
timsaucer
committed
fix(datafusion): reject non-empty children in IcebergPartitionedScan::with_new_children
toutane
committed
fix(datafusion): use ArrowReaderBuilder existing configuration path
toutane
committed
format
toutane
committed
feat(datafusion): bucket FileScanTasks across target_partitions with identity-hash partitioning

authored and
toutane
committed
feat(datafusion): mark identity-partition filters as Exact pushdown

authored and
toutane
committed
feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection

authored and
toutane
committed
Revert "feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection"
toutane
committed
Revert "feat(datafusion): mark identity-partition filters as Exact pushdown"
toutane
committed
refactor(datafusion): merge IcebergPartitionedTableProvider into IcebergTableProvider
toutane
committed
refactor(datafusion): polish scan API and add bucketing tests
toutane
committed

Commits on Apr 30, 2026