chore: update DataFusion dependency to 52.1.0

[timsaucer]

Closes #6009

This PR updates the DataFusion dependency.

It also updates the minimum python version to 3.10 since this is datafusion-python's requirement and 3.9 has reached end of life.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

PR Review: DataFusion 52.1.0 Upgrade

This is a clean dependency upgrade PR. The code changes correctly adapt to DataFusion 52's API changes.

Summary

• DataFusion API adaptations: All changes are straightforward migrations to new API signatures
• Python version bump to 3.10: Reasonable since Python 3.9 reached EOL and datafusion-python requires 3.10
• Test updates: Expected plan output changes due to optimizer behavior differences (removal of CoalesceBatchesExec from plans)

Minor Notes

1. Missing newline at end of file (python/src/lib.rs): The diff shows "No newline at end of file" - consider adding one for consistency.

2. Test pattern change (test_table_provider.py): The change to capture ctx = make_ctx() before calling ctx.table() is correct for ensuring the session context stays alive while the FFI codec is in use.

No P0/P1 issues identified. LGTM.

[wjones127]

I'm working on creating another RC for Lance 3.0.0. If we get this upgrade done in the next few hours, I'm happy to bring this in. Avoids needing to create a major version bump immediately after. I can help with this migration as well.

[timsaucer]

I'm working on creating another RC for Lance 3.0.0. If we get this upgrade done in the next few hours, I'm happy to bring this in. Avoids needing to create a major version bump immediately after. I can help with this migration as well.

Thank you. I am not sure why these tests are failing in the aggregation using avg. I think it's related to this PR but I don't think I'll be able to resolve it in the next few hours. The tests are passing for me locally, which makes it hard to diagnose.

[wjones127]

I'm working on creating another RC for Lance 3.0.0. If we get this upgrade done in the next few hours, I'm happy to bring this in. Avoids needing to create a major version bump immediately after. I can help with this migration as well.

Thank you. I am not sure why these tests are failing in the aggregation using avg. I think it's related to this PR but I don't think I'll be able to resolve it in the next few hours. The tests are passing for me locally, which makes it hard to diagnose.

I think I've fixed that one and am working on fixing the Python tests next. I'll create a PR into your branch when that is ready.

[wjones127]

I think this fixes the remaining tests: rerun-io#11

[wjones127]

Actually moving to #6015

[timsaucer]

Closing in favor of #6015

[timsaucer]

Thanks for picking this up. Sorry, I was slammed this afternoon.