From 64497fd22af59f4b19d7bb0c846f9d8ef4fe2792 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 7 Apr 2026 17:21:28 -0400 Subject: [PATCH 1/3] Update 53 upgrade guide to note release --- .../library-user-guide/upgrading/53.0.0.md | 50 ++++++++++++++++--- 1 file changed, 43 insertions(+), 7 deletions(-) diff --git a/docs/source/library-user-guide/upgrading/53.0.0.md b/docs/source/library-user-guide/upgrading/53.0.0.md index f616220778936..71fe64c4b2b7e 100644 --- a/docs/source/library-user-guide/upgrading/53.0.0.md +++ b/docs/source/library-user-guide/upgrading/53.0.0.md @@ -21,13 +21,6 @@ ## DataFusion 53.0.0 -**Note:** DataFusion `53.0.0` has not been released yet. The information provided -*in this section pertains to features and changes that have already been merged -*to the main branch and are awaiting release in this version. See [#19692] for -\*more details. - -[#19692]: https://github.com/apache/datafusion/issues/19692 - ### Upgrade arrow/parquet to 58.0.0 and object_store to 0.13.0 DataFusion 53.0.0 uses `arrow` and `parquet` 58.0.0, and `object_store` 0.13.0. @@ -39,6 +32,39 @@ See the [Arrow 58.0.0 release notes] and the [object_store 0.13.0 upgrade guide] [arrow 58.0.0 release notes]: https://github.com/apache/arrow-rs/releases/tag/58.0.0 [object_store 0.13.0 upgrade guide]: https://github.com/apache/arrow-rs-object-store/blob/v0.13.0/CHANGELOG.md +### `ExecutionPlan::statistics` removed + +The deprecated `ExecutionPlan::statistics()` method has been removed. If you +implement custom `ExecutionPlan`s, remove that method from your impl and +implement `partition_statistics()` instead. + +**Before:** + +```rust,ignore +impl ExecutionPlan for MyExec { + // ... + + fn statistics(&self) -> Result { + Ok(Statistics::new_unknown(&self.schema())) + } +} +``` + +**After:** + +```rust,ignore +impl ExecutionPlan for MyExec { + // ... + + fn partition_statistics(&self, _partition: Option) -> Result { + Ok(Statistics::new_unknown(&self.schema())) + } +} +``` + +If you do not have partition-specific statistics, return the same value for +`None` and for any partition index. + ### `ExecutionPlan::properties` now returns `&Arc` Now `ExecutionPlan::properties()` returns `&Arc` instead of a @@ -125,6 +151,16 @@ let sub_plan = self.query_to_plan(subquery, planner_context)?; planner_context.pop_outer_query_schema(); ``` +### `HashJoinExec::try_new` adds `null_aware` + +`HashJoinExec::try_new` now takes an extra `null_aware: bool` argument. This +flag is used for null-aware anti joins, such as plans generated for `NOT IN` +subqueries. + +Most callers should pass `false`. Pass `true` only for null-aware +`JoinType::LeftAnti` joins. DataFusion validates that null-aware joins are +single-column `LeftAnti` joins. + ### `FileSinkConfig` adds `file_output_mode` `FileSinkConfig` now includes a `file_output_mode: FileOutputMode` field to control From 20be4a6f343b8498ef30070e9f2d5b583038f7f4 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 9 Apr 2026 07:28:51 -0400 Subject: [PATCH 2/3] Note previous behavior --- docs/source/library-user-guide/upgrading/53.0.0.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/source/library-user-guide/upgrading/53.0.0.md b/docs/source/library-user-guide/upgrading/53.0.0.md index 71fe64c4b2b7e..15e3bfbb876a6 100644 --- a/docs/source/library-user-guide/upgrading/53.0.0.md +++ b/docs/source/library-user-guide/upgrading/53.0.0.md @@ -157,9 +157,8 @@ planner_context.pop_outer_query_schema(); flag is used for null-aware anti joins, such as plans generated for `NOT IN` subqueries. -Most callers should pass `false`. Pass `true` only for null-aware -`JoinType::LeftAnti` joins. DataFusion validates that null-aware joins are -single-column `LeftAnti` joins. +Most callers should pass `false`, the previous behavior. Pass `true` only for null-aware +`JoinType::LeftAnti` joins. ### `FileSinkConfig` adds `file_output_mode` From 8332570737e7fdf278f446b32df9b4cc6667bd25 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 9 Apr 2026 07:29:09 -0400 Subject: [PATCH 3/3] prettier --- docs/source/library-user-guide/upgrading/53.0.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/library-user-guide/upgrading/53.0.0.md b/docs/source/library-user-guide/upgrading/53.0.0.md index 15e3bfbb876a6..f572b2a3f7cc9 100644 --- a/docs/source/library-user-guide/upgrading/53.0.0.md +++ b/docs/source/library-user-guide/upgrading/53.0.0.md @@ -158,7 +158,7 @@ flag is used for null-aware anti joins, such as plans generated for `NOT IN` subqueries. Most callers should pass `false`, the previous behavior. Pass `true` only for null-aware -`JoinType::LeftAnti` joins. +`JoinType::LeftAnti` joins. ### `FileSinkConfig` adds `file_output_mode`