feat: statistics-driven TopK optimization for parquet (file reorder + RG reorder + threshold init + cumulative prune)#21580

Open

zhuqi-lucas wants to merge 38 commits intoapache:mainfrom

zhuqi-lucas:feat/reorder-row-groups-by-stats

Commits on Apr 23, 2026

feat: reorder row groups by statistics during sort pushdown
zhuqi-lucas
committed
test: add SLT tests for row group reorder by statistics
zhuqi-lucas
committed
test: add EXPLAIN assertions for row group reorder tests
zhuqi-lucas
committed
fix: use max statistics for DESC sort reorder
zhuqi-lucas
committed
fix: prevent reorder+reverse double-reordering of row groups
zhuqi-lucas
committed
fix: rebase conflicts and compilation errors
zhuqi-lucas
committed
refactor: introduce AccessPlanOptimizer trait for row group reordering
zhuqi-lucas
committed
chore: remove benchmark from this PR (tracked in #21582)
zhuqi-lucas
committed
fix: resolve doc link for AccessPlanOptimizer
zhuqi-lucas
committed
fix: restore benchmark files from upstream main
zhuqi-lucas
committed
fix: compose reorder and reverse as sequential steps instead of mutually exclusive
zhuqi-lucas
committed
fix: generate scrambled+overlapping RGs for overlap benchmark
zhuqi-lucas
committed
feat: reorder files in shared work queue by statistics for TopK
zhuqi-lucas
committed
feat: initialize TopK dynamic filter threshold from parquet statistics
zhuqi-lucas
committed
feat: enable file reorder and RG reorder for all TopK queries
zhuqi-lucas
committed
perf: move stats init before RG pruning so first file also benefits
zhuqi-lucas
committed
fix: restrict RG reorder/reverse to sort pushdown path only
zhuqi-lucas
committed
perf: move stats init before PruningPredicate build + fix CastExpr unwrap
zhuqi-lucas
committed
fix: null-aware filter + restrict stats init to sort pushdown path
zhuqi-lucas
committed
feat: enable stats init for ALL TopK queries + fix fuzz test tiebreaker
zhuqi-lucas
committed
fix: restrict stats init to sort pushdown path to avoid over-pruning
zhuqi-lucas
committed
fix: stats init only safe for sorted (non-overlapping) RGs
zhuqi-lucas
committed
feat: enable stats init for pure TopK queries (no WHERE clause)
zhuqi-lucas
committed
fix: stats init requires sort pushdown + no WHERE clause
zhuqi-lucas
committed
feat: TopK cumulative RG pruning after reorder (works with WHERE)
zhuqi-lucas
committed
feat: enable RG reorder + cumulative prune for all TopK queries
zhuqi-lucas
committed
fix: only reverse/cumulate when reorder succeeds (prevents ClickBench regression)
zhuqi-lucas
committed
fix: escape brackets in doc comment to fix rustdoc link error
zhuqi-lucas
committed
chore: remove benchmark and listing_table_partitions changes from this PR
zhuqi-lucas
committed
refactor: remove stats init in favor of cumulative RG pruning + add SLT tests
zhuqi-lucas
committed
fix: cumulative prune only without WHERE to avoid under-returning rows
zhuqi-lucas
committed
feat: restore stats init with fixes (GtEq + df.fetch() + type cast)
zhuqi-lucas
committed
fix: SortExec.fetch was 0 when create_filter was called
zhuqi-lucas
committed
perf: skip RG reorder when sort column not in file schema
zhuqi-lucas
committed
fix: use slt:ignore for non-deterministic output_rows_skew metric
zhuqi-lucas
committed
fix: skip cumulative prune when row_selection exists + cleanup
zhuqi-lucas
committed
feat: support multi-key ORDER BY + truncate row_selection safety
zhuqi-lucas
committed
test: add multi-key Inexact sort pushdown test (Test M)
zhuqi-lucas
committed