Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 42 additions & 7 deletions docs/source/library-user-guide/upgrading/53.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,6 @@

## DataFusion 53.0.0

**Note:** DataFusion `53.0.0` has not been released yet. The information provided
*in this section pertains to features and changes that have already been merged
*to the main branch and are awaiting release in this version. See [#19692] for
\*more details.

[#19692]: https://github.com/apache/datafusion/issues/19692

### Upgrade arrow/parquet to 58.0.0 and object_store to 0.13.0

DataFusion 53.0.0 uses `arrow` and `parquet` 58.0.0, and `object_store` 0.13.0.
Expand All @@ -39,6 +32,39 @@ See the [Arrow 58.0.0 release notes] and the [object_store 0.13.0 upgrade guide]
[arrow 58.0.0 release notes]: https://github.com/apache/arrow-rs/releases/tag/58.0.0
[object_store 0.13.0 upgrade guide]: https://github.com/apache/arrow-rs-object-store/blob/v0.13.0/CHANGELOG.md

### `ExecutionPlan::statistics` removed

The deprecated `ExecutionPlan::statistics()` method has been removed. If you
implement custom `ExecutionPlan`s, remove that method from your impl and
implement `partition_statistics()` instead.

**Before:**

```rust,ignore
impl ExecutionPlan for MyExec {
// ...

fn statistics(&self) -> Result<Statistics> {
Ok(Statistics::new_unknown(&self.schema()))
}
}
```

**After:**

```rust,ignore
impl ExecutionPlan for MyExec {
// ...

fn partition_statistics(&self, _partition: Option<usize>) -> Result<Statistics> {
Ok(Statistics::new_unknown(&self.schema()))
}
}
```

If you do not have partition-specific statistics, return the same value for
`None` and for any partition index.

### `ExecutionPlan::properties` now returns `&Arc<PlanProperties>`

Now `ExecutionPlan::properties()` returns `&Arc<PlanProperties>` instead of a
Expand Down Expand Up @@ -125,6 +151,15 @@ let sub_plan = self.query_to_plan(subquery, planner_context)?;
planner_context.pop_outer_query_schema();
```

### `HashJoinExec::try_new` adds `null_aware`

`HashJoinExec::try_new` now takes an extra `null_aware: bool` argument. This
flag is used for null-aware anti joins, such as plans generated for `NOT IN`
subqueries.

Most callers should pass `false`, the previous behavior. Pass `true` only for null-aware
`JoinType::LeftAnti` joins.

### `FileSinkConfig` adds `file_output_mode`

`FileSinkConfig` now includes a `file_output_mode: FileOutputMode` field to control
Expand Down
Loading