Skip to content

chore: upgrade datafusion to 51, arrow to 57, pyo3 to 0.26#5291

Merged
wjones127 merged 12 commits intolance-format:mainfrom
rerun-io:tsaucer/df51
Jan 7, 2026
Merged

chore: upgrade datafusion to 51, arrow to 57, pyo3 to 0.26#5291
wjones127 merged 12 commits intolance-format:mainfrom
rerun-io:tsaucer/df51

Conversation

@timsaucer
Copy link
Copy Markdown
Contributor

@timsaucer timsaucer commented Nov 19, 2025 •
edited
Loading

Closes #5302

This PR is a dependency update PR. The notable updates are:

  • Arrow to 57
  • DataFusion to 51
  • Pyo3 to 0.26

Most of the churn in this update is related to Pyo3, which had a few APIs deprecated.

Caveat: Large portions of this PR were written by Claude, but I have reviewed the entire PR.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Nov 19, 2025

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@timsaucer timsaucer changed the title Upgrade datafusion to 51, arrow to 57, pyo3 to 0.26 chore: upgrade datafusion to 51, arrow to 57, pyo3 to 0.26 Nov 19, 2025
@github-actions github-actions Bot added the chore label Nov 19, 2025
@timsaucer
Copy link
Copy Markdown
Contributor Author

timsaucer commented Nov 21, 2025

@westonpace In order to build, one of the things I'm finding is a need to update prost / substrait. It looks like substrait-expr is a personal repo of yours. I had an agent take a first stab at updating it, but there are a couple of things that need resolving before it's ready to go. I'm adding a note here for tracking purposes: westonpace/substrait-expr#28

@ddupg ddupg mentioned this pull request Nov 27, 2025
Closed
3 tasks
@timsaucer
Copy link
Copy Markdown
Contributor Author

timsaucer commented Dec 2, 2025

This is currently blocked by the new dependency on geoarrow and geodatafusion. We need those to upgrade to Arrow 57 and DataFusion 50 respectively.

@westonpace
Copy link
Copy Markdown
Member

westonpace commented Dec 3, 2025

Looks like go-arrow needs arrow-wasm to bump 😆 😭

A c-data version-agnostic API for lance is looking more and more appealing everyday.

@wjones127 wjones127 mentioned this pull request Dec 21, 2025
Open
@valkum
Copy link
Copy Markdown
Contributor

valkum commented Jan 6, 2026

The respective PR in geodatafusion is datafusion-contrib/geodatafusion#50. geoarrow 0.7 was released yesterday.

@github-actions github-actions Bot added the java label Jan 7, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 7, 2026

Codecov Report

❌ Patch coverage is 40.42553% with 28 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-index/src/scalar/btree.rs 0.00% 18 Missing and 2 partials ⚠️
rust/lance/src/dataset/scanner.rs 36.36% 3 Missing and 4 partials ⚠️
rust/lance/src/dataset/write/merge_insert.rs 50.00% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@timsaucer timsaucer marked this pull request as ready for review January 7, 2026 21:39
@timsaucer
Copy link
Copy Markdown
Contributor Author

timsaucer commented Jan 7, 2026

CI is in the green! @westonpace it looks like we're ready to upgrade DataFusion & Arrow!

wjones127
wjones127 reviewed Jan 7, 2026
View reviewed changes
Comment thread rust/lance-datafusion/src/exec.rs

let schema = plan.schema();
let analyze = Arc::new(AnalyzeExec::new(true, true, plan, schema));
// TODO(tsaucer) I chose SUMMARY here but do we also want DEV?
Copy link
Copy Markdown
Contributor

@wjones127 wjones127 Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine for now.

wjones127
wjones127 approved these changes Jan 7, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@wjones127 wjones127 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for doing the PyO3 stuff. That could not have been fun.

I'm not sure what to make of this, but DataFusion's MSRV is supposedly 1.88 now, but our MSRV is at 1.82 and the CI check is still passing. 🤷

@wjones127
Copy link
Copy Markdown
Contributor

wjones127 commented Jan 7, 2026

Ah it looks like the upgrade to MSRV is something we can do independently to enable using the 2024 edition in Rust. We might do that soon. :)

@wjones127 wjones127 merged commit 18587ea into lance-format:main Jan 7, 2026
33 checks passed
@timsaucer
Copy link
Copy Markdown
Contributor Author

timsaucer commented Jan 7, 2026

Ah it looks like the upgrade to MSRV is something we can do independently to enable using the 2024 edition in Rust. We might do that soon. :)

The let chains in 2024 edition are very nice.

@timsaucer timsaucer deleted the tsaucer/df51 branch January 7, 2026 23:04
@westonpace
Copy link
Copy Markdown
Member

westonpace commented Jan 7, 2026

Thanks for staying on top of this!

@gotocoding-DB
Copy link
Copy Markdown

gotocoding-DB commented Jan 13, 2026

@timsaucer Just came to write and say thank you very much for updating the arrow!

jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026
@timsaucer @jackye1995
chore: upgrade datafusion to 51, arrow to 57, pyo3 to 0.26 (lance-for…
69344d0 
…mat#5291)

Closes lance-format#5302

This PR is a dependency update PR. The notable updates are:

- Arrow to 57
- DataFusion to 51
- Pyo3 to 0.26

Most of the churn in this update is related to Pyo3, which had a few
APIs deprecated.

Caveat: Large portions of this PR were written by Claude, but I have
reviewed the entire PR.
@gotocoding-DB gotocoding-DB mentioned this pull request Mar 9, 2026
Open
@andrea-reale andrea-reale mentioned this pull request Mar 30, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wjones127 wjones127 wjones127 approved these changes

Assignees

No one assigned

Labels

chore java python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Upgrade to DataFusion 51 and Arrow 57

5 participants

@timsaucer @westonpace @valkum @wjones127 @gotocoding-DB