Remove coalesce batches rule and deprecate CoalesceBatchesExec by feniljain · Pull Request #19622 · apache/datafusion

feniljain · 2026-01-03T11:57:21Z

Which issue does this PR close?

Closes Remove CoalesceBatches optimizer rule #19591

Rationale for this change

Explained in issue itself

What changes are included in this PR?

Removed coalesce batches rule
Deprecate CoalesceBatchesExec

Are these changes tested?

Yes

Are there any user-facing changes?

Yes, added a deprecation tag on CoalesceBatchesExec

CoalesceBatchesExec

Dandandan · 2026-01-03T12:41:38Z

Looks great, can you fix the clippy errors @feniljain ?

Dandandan · 2026-01-03T12:57:45Z

run benchmarks

alamb-ghbot · 2026-01-03T13:09:11Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing feat-deprecate-coalesce-batches (49b2473) to 70daf88 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-03T13:49:22Z

🤖: Benchmark completed

Details

Comparing HEAD and feat-deprecate-coalesce-batches
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2480.64 ms │                      2376.12 ms │     no change │
│ QQuery 1     │   922.24 ms │                       957.71 ms │     no change │
│ QQuery 2     │  1862.91 ms │                      1879.40 ms │     no change │
│ QQuery 3     │  1221.40 ms │                      1125.58 ms │ +1.09x faster │
│ QQuery 4     │  2301.57 ms │                      2362.44 ms │     no change │
│ QQuery 5     │ 28157.18 ms │                     28362.29 ms │     no change │
│ QQuery 6     │  4019.51 ms │                      4046.59 ms │     no change │
│ QQuery 7     │  3773.09 ms │                      3755.00 ms │     no change │
└──────────────┴─────────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 44738.53ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 44865.13ms │
│ Average Time (HEAD)                            │  5592.32ms │
│ Average Time (feat-deprecate-coalesce-batches) │  5608.14ms │
│ Queries Faster                                 │          1 │
│ Queries Slower                                 │          0 │
│ Queries with No Change                         │          7 │
│ Queries with Failure                           │          0 │
└────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.42 ms │                         1.41 ms │     no change │
│ QQuery 1     │    49.57 ms │                        49.97 ms │     no change │
│ QQuery 2     │   133.27 ms │                       132.94 ms │     no change │
│ QQuery 3     │   153.53 ms │                       151.33 ms │     no change │
│ QQuery 4     │  1115.17 ms │                      1112.62 ms │     no change │
│ QQuery 5     │  1409.93 ms │                      1401.43 ms │     no change │
│ QQuery 6     │     1.49 ms │                         1.45 ms │     no change │
│ QQuery 7     │    55.43 ms │                        56.65 ms │     no change │
│ QQuery 8     │  1457.94 ms │                      1453.06 ms │     no change │
│ QQuery 9     │  1860.84 ms │                      1830.20 ms │     no change │
│ QQuery 10    │   345.30 ms │                       342.50 ms │     no change │
│ QQuery 11    │   394.20 ms │                       399.11 ms │     no change │
│ QQuery 12    │  1284.94 ms │                      1258.95 ms │     no change │
│ QQuery 13    │  1987.77 ms │                      1970.92 ms │     no change │
│ QQuery 14    │  1246.54 ms │                      1243.64 ms │     no change │
│ QQuery 15    │  1253.03 ms │                      1238.85 ms │     no change │
│ QQuery 16    │  2559.88 ms │                      2539.36 ms │     no change │
│ QQuery 17    │  2510.75 ms │                      2563.07 ms │     no change │
│ QQuery 18    │  5205.59 ms │                      4968.51 ms │     no change │
│ QQuery 19    │   123.04 ms │                       122.06 ms │     no change │
│ QQuery 20    │  1873.75 ms │                      1888.64 ms │     no change │
│ QQuery 21    │  2214.55 ms │                      2182.75 ms │     no change │
│ QQuery 22    │  3792.25 ms │                      3735.36 ms │     no change │
│ QQuery 23    │ 24628.13 ms │                     12220.70 ms │ +2.02x faster │
│ QQuery 24    │   226.36 ms │                       203.98 ms │ +1.11x faster │
│ QQuery 25    │   479.66 ms │                       463.33 ms │     no change │
│ QQuery 26    │   220.61 ms │                       201.84 ms │ +1.09x faster │
│ QQuery 27    │  2843.16 ms │                      2689.96 ms │ +1.06x faster │
│ QQuery 28    │ 23837.99 ms │                     23277.01 ms │     no change │
│ QQuery 29    │   967.37 ms │                       957.80 ms │     no change │
│ QQuery 30    │  1341.13 ms │                      1305.74 ms │     no change │
│ QQuery 31    │  1367.97 ms │                      1319.54 ms │     no change │
│ QQuery 32    │  4966.06 ms │                      5006.32 ms │     no change │
│ QQuery 33    │  5731.50 ms │                      5455.90 ms │     no change │
│ QQuery 34    │  5800.78 ms │                      5718.56 ms │     no change │
│ QQuery 35    │  2010.41 ms │                      1884.98 ms │ +1.07x faster │
│ QQuery 36    │    64.40 ms │                        65.17 ms │     no change │
│ QQuery 37    │    43.30 ms │                        43.81 ms │     no change │
│ QQuery 38    │    66.83 ms │                        63.76 ms │     no change │
│ QQuery 39    │    99.23 ms │                       103.67 ms │     no change │
│ QQuery 40    │    27.19 ms │                        27.49 ms │     no change │
│ QQuery 41    │    22.98 ms │                        21.67 ms │ +1.06x faster │
│ QQuery 42    │    20.15 ms │                        19.40 ms │     no change │
└──────────────┴─────────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 105795.39ms │
│ Total Time (feat-deprecate-coalesce-batches)   │  91695.40ms │
│ Average Time (HEAD)                            │   2460.36ms │
│ Average Time (feat-deprecate-coalesce-batches) │   2132.45ms │
│ Queries Faster                                 │           6 │
│ Queries Slower                                 │           0 │
│ Queries with No Change                         │          37 │
│ Queries with Failure                           │           0 │
└────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 120.57 ms │                       116.85 ms │     no change │
│ QQuery 2     │  26.48 ms │                        28.38 ms │  1.07x slower │
│ QQuery 3     │  36.42 ms │                        33.73 ms │ +1.08x faster │
│ QQuery 4     │  28.42 ms │                        29.75 ms │     no change │
│ QQuery 5     │  85.92 ms │                        88.10 ms │     no change │
│ QQuery 6     │  19.63 ms │                        19.83 ms │     no change │
│ QQuery 7     │ 228.15 ms │                       225.51 ms │     no change │
│ QQuery 8     │  35.66 ms │                        33.02 ms │ +1.08x faster │
│ QQuery 9     │ 117.08 ms │                       101.27 ms │ +1.16x faster │
│ QQuery 10    │  64.26 ms │                        63.05 ms │     no change │
│ QQuery 11    │  16.75 ms │                        18.22 ms │  1.09x slower │
│ QQuery 12    │  50.39 ms │                        50.21 ms │     no change │
│ QQuery 13    │  46.44 ms │                        47.12 ms │     no change │
│ QQuery 14    │  13.20 ms │                        13.89 ms │  1.05x slower │
│ QQuery 15    │  23.82 ms │                        24.57 ms │     no change │
│ QQuery 16    │  23.85 ms │                        24.75 ms │     no change │
│ QQuery 17    │ 150.27 ms │                       149.34 ms │     no change │
│ QQuery 18    │ 278.44 ms │                       288.25 ms │     no change │
│ QQuery 19    │  36.51 ms │                        38.58 ms │  1.06x slower │
│ QQuery 20    │  47.58 ms │                        51.71 ms │  1.09x slower │
│ QQuery 21    │ 329.56 ms │                       322.19 ms │     no change │
│ QQuery 22    │  17.53 ms │                        17.84 ms │     no change │
└──────────────┴───────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 1796.91ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 1786.18ms │
│ Average Time (HEAD)                            │   81.68ms │
│ Average Time (feat-deprecate-coalesce-batches) │   81.19ms │
│ Queries Faster                                 │         3 │
│ Queries Slower                                 │         5 │
│ Queries with No Change                         │        14 │
│ Queries with Failure                           │         0 │
└────────────────────────────────────────────────┴───────────┘

feniljain · 2026-01-03T14:03:30Z

QQuery 23 in clickbench_partitioned just halved the time 😲

Dandandan · 2026-01-03T14:15:37Z

QQuery 23 in clickbench_partitioned just halved the time 😲

I think this is not correct, it is super flaky lately. @alamb do you know the source of this? Is it running OOM / swapping perhaps or slow / not predictable block storage?

feniljain · 2026-01-03T14:16:50Z

I think this is not correct, it is super flaky lately.

Ouuh

@ alamb-ghbot

Did you mean to tag Andrew instead of bot?

Dandandan · 2026-01-03T14:33:53Z

I think this is not correct, it is super flaky lately.

Ouuh

@ alamb-ghbot

Did you mean to tag Andrew instead of bot?

Yeah @alamb :D

Dandandan · 2026-01-03T14:34:22Z

run benchmarks

alamb-ghbot · 2026-01-03T15:22:26Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing feat-deprecate-coalesce-batches (7d844f6) to 70daf88 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-03T15:59:50Z

🤖: Benchmark completed

Details

Comparing HEAD and feat-deprecate-coalesce-batches
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2500.64 ms │                      2482.51 ms │     no change │
│ QQuery 1     │   973.64 ms │                       954.69 ms │     no change │
│ QQuery 2     │  1907.80 ms │                      1867.79 ms │     no change │
│ QQuery 3     │  1171.75 ms │                      1149.00 ms │     no change │
│ QQuery 4     │  2292.04 ms │                      2302.01 ms │     no change │
│ QQuery 5     │ 28668.58 ms │                     28370.85 ms │     no change │
│ QQuery 6     │  4006.33 ms │                      4025.76 ms │     no change │
│ QQuery 7     │  3880.19 ms │                      3570.73 ms │ +1.09x faster │
└──────────────┴─────────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 45400.97ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 44723.34ms │
│ Average Time (HEAD)                            │  5675.12ms │
│ Average Time (feat-deprecate-coalesce-batches) │  5590.42ms │
│ Queries Faster                                 │          1 │
│ Queries Slower                                 │          0 │
│ Queries with No Change                         │          7 │
│ Queries with Failure                           │          0 │
└────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.41 ms │                         1.43 ms │     no change │
│ QQuery 1     │    49.83 ms │                        49.78 ms │     no change │
│ QQuery 2     │   134.79 ms │                       132.45 ms │     no change │
│ QQuery 3     │   159.22 ms │                       154.67 ms │     no change │
│ QQuery 4     │  1204.05 ms │                      1113.18 ms │ +1.08x faster │
│ QQuery 5     │  1540.13 ms │                      1387.51 ms │ +1.11x faster │
│ QQuery 6     │     1.44 ms │                         1.41 ms │     no change │
│ QQuery 7     │    54.82 ms │                        56.59 ms │     no change │
│ QQuery 8     │  1561.22 ms │                      1466.54 ms │ +1.06x faster │
│ QQuery 9     │  1918.42 ms │                      1848.28 ms │     no change │
│ QQuery 10    │   355.38 ms │                       343.33 ms │     no change │
│ QQuery 11    │   409.97 ms │                       391.06 ms │     no change │
│ QQuery 12    │  1380.74 ms │                      1314.80 ms │     no change │
│ QQuery 13    │  2076.82 ms │                      1982.58 ms │     no change │
│ QQuery 14    │  1365.16 ms │                      1259.13 ms │ +1.08x faster │
│ QQuery 15    │  1338.03 ms │                      1279.45 ms │     no change │
│ QQuery 16    │  2695.83 ms │                      2591.58 ms │     no change │
│ QQuery 17    │  2634.26 ms │                      2580.71 ms │     no change │
│ QQuery 18    │  5512.12 ms │                      4958.17 ms │ +1.11x faster │
│ QQuery 19    │   120.59 ms │                       120.51 ms │     no change │
│ QQuery 20    │  1937.78 ms │                      1899.25 ms │     no change │
│ QQuery 21    │  2205.14 ms │                      2206.84 ms │     no change │
│ QQuery 22    │  3861.19 ms │                      3791.22 ms │     no change │
│ QQuery 23    │ 17397.06 ms │                     12255.24 ms │ +1.42x faster │
│ QQuery 24    │   205.63 ms │                       211.23 ms │     no change │
│ QQuery 25    │   469.96 ms │                       468.71 ms │     no change │
│ QQuery 26    │   209.65 ms │                       215.96 ms │     no change │
│ QQuery 27    │  2818.25 ms │                      2741.69 ms │     no change │
│ QQuery 28    │ 23529.40 ms │                     23554.26 ms │     no change │
│ QQuery 29    │   978.43 ms │                       975.87 ms │     no change │
│ QQuery 30    │  1373.00 ms │                      1340.90 ms │     no change │
│ QQuery 31    │  1385.89 ms │                      1343.46 ms │     no change │
│ QQuery 32    │  5017.94 ms │                      4831.39 ms │     no change │
│ QQuery 33    │  5707.72 ms │                      5579.63 ms │     no change │
│ QQuery 34    │  5813.03 ms │                      5777.31 ms │     no change │
│ QQuery 35    │  2013.03 ms │                      1939.79 ms │     no change │
│ QQuery 36    │    67.00 ms │                        64.95 ms │     no change │
│ QQuery 37    │    45.30 ms │                        46.18 ms │     no change │
│ QQuery 38    │    65.47 ms │                        65.38 ms │     no change │
│ QQuery 39    │    99.28 ms │                       102.11 ms │     no change │
│ QQuery 40    │    25.94 ms │                        25.14 ms │     no change │
│ QQuery 41    │    22.92 ms │                        22.69 ms │     no change │
│ QQuery 42    │    19.84 ms │                        19.30 ms │     no change │
└──────────────┴─────────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 99783.09ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 92511.69ms │
│ Average Time (HEAD)                            │  2320.54ms │
│ Average Time (feat-deprecate-coalesce-batches) │  2151.43ms │
│ Queries Faster                                 │          6 │
│ Queries Slower                                 │          0 │
│ Queries with No Change                         │         37 │
│ Queries with Failure                           │          0 │
└────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ feat-deprecate-coalesce-batches ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 119.23 ms │                       117.53 ms │    no change │
│ QQuery 2     │  29.02 ms │                        28.25 ms │    no change │
│ QQuery 3     │  37.36 ms │                        37.92 ms │    no change │
│ QQuery 4     │  28.40 ms │                        29.35 ms │    no change │
│ QQuery 5     │  85.59 ms │                        87.76 ms │    no change │
│ QQuery 6     │  19.77 ms │                        19.48 ms │    no change │
│ QQuery 7     │ 223.14 ms │                       223.93 ms │    no change │
│ QQuery 8     │  31.65 ms │                        36.57 ms │ 1.16x slower │
│ QQuery 9     │  97.87 ms │                       101.92 ms │    no change │
│ QQuery 10    │  60.70 ms │                        64.59 ms │ 1.06x slower │
│ QQuery 11    │  17.70 ms │                        19.06 ms │ 1.08x slower │
│ QQuery 12    │  50.25 ms │                        49.68 ms │    no change │
│ QQuery 13    │  47.32 ms │                        47.48 ms │    no change │
│ QQuery 14    │  13.15 ms │                        13.40 ms │    no change │
│ QQuery 15    │  23.97 ms │                        23.95 ms │    no change │
│ QQuery 16    │  23.82 ms │                        24.26 ms │    no change │
│ QQuery 17    │ 147.00 ms │                       150.02 ms │    no change │
│ QQuery 18    │ 269.21 ms │                       276.40 ms │    no change │
│ QQuery 19    │  36.99 ms │                        36.69 ms │    no change │
│ QQuery 20    │  50.71 ms │                        50.94 ms │    no change │
│ QQuery 21    │ 293.05 ms │                       301.26 ms │    no change │
│ QQuery 22    │  17.21 ms │                        17.57 ms │    no change │
└──────────────┴───────────┴─────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 1723.11ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 1758.02ms │
│ Average Time (HEAD)                            │   78.32ms │
│ Average Time (feat-deprecate-coalesce-batches) │   79.91ms │
│ Queries Faster                                 │         0 │
│ Queries Slower                                 │         3 │
│ Queries with No Change                         │        19 │
│ Queries with Failure                           │         0 │
└────────────────────────────────────────────────┴───────────┘

feniljain · 2026-01-03T16:18:18Z

QQuery 23 still seems to be leading ahead!

Dandandan · 2026-01-03T16:37:37Z

QQuery 23 still seems to be leading ahead!

I think it is the same flakiness

datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs

…ariants

alamb · 2026-01-05T20:00:45Z

QQuery 23 still seems to be leading ahead!

I suspect this has to do with timing. Basically Q23 is like select * from ... WHERE ... type query

This can now takes advantage of dynamic filter pushdown which can prune some of the files. However, the updates of the dynamic filters is not deterministic so I think that accounts for the variance run to run -- depending on what rows have been seen already will potentially skip entire files

I suspect this accounts for the non determinsim

I will see if I can reproduce the results locally

alamb · 2026-01-05T20:00:58Z

run benchmark clickbench_partitioned

alamb-ghbot · 2026-01-05T20:01:07Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing feat-deprecate-coalesce-batches (43f30c2) to 70daf88 diff using: clickbench_partitioned
Results will be posted here when complete

alamb

Thank you @feniljain -- this looks great to me

datafusion/physical-plan/src/coalesce_batches.rs

datafusion/core/tests/physical_optimizer/pushdown_sort.rs

alamb · 2026-01-05T20:05:02Z

datafusion/core/tests/physical_optimizer/replace_with_order_preserving_variants.rs

@@ -440,9 +439,7 @@ async fn test_replace_multiple_input_repartition_with_extra_steps(
    let repartition_rr = repartition_exec_round_robin(source);
    let repartition_hash = repartition_exec_hash(repartition_rr);


not for this PR, but I think now actually we have fixed the double repartition as well, as described in https://datafusion.apache.org/blog/output/2025/12/15/avoid-consecutive-repartitions/

@gene-bordegaray is that correct? if so I will file a ticket to try and update these tests so they match what datafusion creates now (a single round robin hash)

datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs

alamb-ghbot · 2026-01-05T20:31:54Z

🤖: Benchmark completed

Details

Comparing HEAD and feat-deprecate-coalesce-batches
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ feat-deprecate-coalesce-batches ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.39 ms │                         1.43 ms │     no change │
│ QQuery 1     │    48.82 ms │                        51.70 ms │  1.06x slower │
│ QQuery 2     │   129.60 ms │                       137.01 ms │  1.06x slower │
│ QQuery 3     │   155.19 ms │                       149.54 ms │     no change │
│ QQuery 4     │  1074.95 ms │                      1067.25 ms │     no change │
│ QQuery 5     │  1377.60 ms │                      1366.89 ms │     no change │
│ QQuery 6     │     1.44 ms │                         1.43 ms │     no change │
│ QQuery 7     │    52.63 ms │                        55.75 ms │  1.06x slower │
│ QQuery 8     │  1411.58 ms │                      1457.04 ms │     no change │
│ QQuery 9     │  1837.05 ms │                      1776.04 ms │     no change │
│ QQuery 10    │   339.27 ms │                       346.48 ms │     no change │
│ QQuery 11    │   394.35 ms │                       391.15 ms │     no change │
│ QQuery 12    │  1264.90 ms │                      1280.20 ms │     no change │
│ QQuery 13    │  1938.67 ms │                      1983.32 ms │     no change │
│ QQuery 14    │  1237.96 ms │                      1242.78 ms │     no change │
│ QQuery 15    │  1246.66 ms │                      1246.40 ms │     no change │
│ QQuery 16    │  2581.38 ms │                      2537.25 ms │     no change │
│ QQuery 17    │  2553.27 ms │                      2540.27 ms │     no change │
│ QQuery 18    │  5422.77 ms │                      4828.64 ms │ +1.12x faster │
│ QQuery 19    │   122.23 ms │                       119.88 ms │     no change │
│ QQuery 20    │  1912.41 ms │                      1899.88 ms │     no change │
│ QQuery 21    │  2172.87 ms │                      2188.42 ms │     no change │
│ QQuery 22    │  3784.37 ms │                      3761.73 ms │     no change │
│ QQuery 23    │ 12377.60 ms │                     12180.90 ms │     no change │
│ QQuery 24    │   210.08 ms │                       207.88 ms │     no change │
│ QQuery 25    │   464.79 ms │                       464.33 ms │     no change │
│ QQuery 26    │   212.33 ms │                       212.99 ms │     no change │
│ QQuery 27    │  2764.08 ms │                      2725.10 ms │     no change │
│ QQuery 28    │ 23251.56 ms │                     23602.78 ms │     no change │
│ QQuery 29    │   948.05 ms │                       936.18 ms │     no change │
│ QQuery 30    │  1363.55 ms │                      1304.87 ms │     no change │
│ QQuery 31    │  1388.14 ms │                      1340.96 ms │     no change │
│ QQuery 32    │  4793.19 ms │                      5131.87 ms │  1.07x slower │
│ QQuery 33    │  5489.43 ms │                      5730.76 ms │     no change │
│ QQuery 34    │  5789.37 ms │                      6284.25 ms │  1.09x slower │
│ QQuery 35    │  1993.99 ms │                      1923.49 ms │     no change │
│ QQuery 36    │    67.73 ms │                        69.05 ms │     no change │
│ QQuery 37    │    44.92 ms │                        44.70 ms │     no change │
│ QQuery 38    │    66.05 ms │                        65.74 ms │     no change │
│ QQuery 39    │   101.33 ms │                       104.03 ms │     no change │
│ QQuery 40    │    26.76 ms │                        25.55 ms │     no change │
│ QQuery 41    │    22.27 ms │                        22.12 ms │     no change │
│ QQuery 42    │    19.12 ms │                        19.01 ms │     no change │
└──────────────┴─────────────┴─────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                              ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                              │ 92455.66ms │
│ Total Time (feat-deprecate-coalesce-batches)   │ 92827.05ms │
│ Average Time (HEAD)                            │  2150.13ms │
│ Average Time (feat-deprecate-coalesce-batches) │  2158.77ms │
│ Queries Faster                                 │          1 │
│ Queries Slower                                 │          5 │
│ Queries with No Change                         │         37 │
│ Queries with Failure                           │          0 │
└────────────────────────────────────────────────┴────────────┘

alamb · 2026-01-06T20:55:38Z

Epic work @feniljain -- thanks again

…e#19622) ## Which issue does this PR close? - Closes apache#19591 ## Rationale for this change Explained in issue itself ## What changes are included in this PR? - Removed coalesce batches rule - Deprecate `CoalesceBatchesExec` ## Are these changes tested? Yes ## Are there any user-facing changes? Yes, added a deprecation tag on `CoalesceBatchesExec`

feat: remove coalesce batches optimizer rule and deprecate

7b30070

CoalesceBatchesExec

github-actions bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jan 3, 2026

feniljain added 3 commits January 3, 2026 17:32

chore: cargo fmt

9b5f438

chore(clippy): add expect deprecated tag

2f1e6a2

test: update async udf example test

49b2473

github-actions bot added the proto Related to proto crate label Jan 3, 2026

chore(clippy): add expect deprecated tag for all targets

7d844f6

github-actions bot added the core Core DataFusion crate label Jan 3, 2026

Dandandan reviewed Jan 3, 2026

View reviewed changes

datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs Outdated Show resolved Hide resolved

feniljain added 6 commits January 4, 2026 21:49

test: remove coalescebatchesexec from coop test

9ad00b3

test: remove coalescebatchesexec from filter_pushdown test

31f4dd1

test: remove coalescebatchesexec from enforce_distribution

9ecc741

test: remove coalescebatchesexec from partition_statistics

3d6928e

test: remove coalescebatchesexec from projection_pushdown

1f9c261

test: remove coalescebatchesexec from replace_with_order_preserving_v…

ed2a643

…ariants

feniljain added 7 commits January 4, 2026 22:43

test: remove coalescebatchesexec from explain_analyze

c25a897

test: remove coalescebatchesexec from aggregates

33cff4b

test: remove coalescebatchesexec from sort_preserving_merge

166b335

test: remove coalescebatchesexec from enforce_sorting

4d6d5d4

test: remove coalescebatchesexec from limit_pushdown

850026a

test: remove coalescebatchesexec from pushdown_sort

4e5eb61

test: remove coalescebatchesexec from test_utils

43f30c2

alamb approved these changes Jan 5, 2026

View reviewed changes

alamb added this pull request to the merge queue Jan 6, 2026

Merged via the queue into apache:main with commit e8196f4 Jan 6, 2026
32 checks passed

alamb mentioned this pull request Jan 6, 2026

[EPIC] Remove CoalesceBatchesExec operator #18779

Closed

12 tasks

xudong963 mentioned this pull request Feb 9, 2026

Cherry pick push down join key filters for LEFT/RIGHT/ANTI joins massive-com/arrow-datafusion#32

Merged

		@@ -440,9 +439,7 @@ async fn test_replace_multiple_input_repartition_with_extra_steps(
		let repartition_rr = repartition_exec_round_robin(source);
		let repartition_hash = repartition_exec_hash(repartition_rr);

Conversation

feniljain commented Jan 3, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Dandandan commented Jan 3, 2026

Uh oh!

Dandandan commented Jan 3, 2026

Uh oh!

alamb-ghbot commented Jan 3, 2026

Uh oh!

alamb-ghbot commented Jan 3, 2026

Uh oh!

feniljain commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

feniljain commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Jan 3, 2026

Uh oh!

Dandandan commented Jan 3, 2026

Uh oh!

alamb-ghbot commented Jan 3, 2026

Uh oh!

alamb-ghbot commented Jan 3, 2026

Uh oh!

feniljain commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Jan 3, 2026

Uh oh!

Uh oh!

alamb commented Jan 5, 2026

Uh oh!

alamb commented Jan 5, 2026

Uh oh!

alamb-ghbot commented Jan 5, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb-ghbot commented Jan 5, 2026

Uh oh!

alamb commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feniljain commented Jan 3, 2026 •

edited

Loading

Dandandan commented Jan 3, 2026 •

edited

Loading

feniljain commented Jan 3, 2026 •

edited

Loading

feniljain commented Jan 3, 2026 •

edited

Loading