Tracking Upgrade to Datafusion 37

[Michael-J-Ward]

I sketched out some of the upgrade in #662 and wanted to share what I encountered.

Some major changes in between datafusion 36 and 37.

Needs Completing

• 1) Porting functions.rs BuiltInFunctons to datafusion-functions
• 2) Replacing SectionContext::tables or updating the tests to not rely on it
• 3) new trait methods for ExecutionPlan and ExecutionPlanProperties for DatasetExec need implementing

1) Porting BuiltinFunctions

See this epic apache/datafusion#9285.

That means much of datafusion-python::functions.rs needs an update. An incomplete first pass is 83249fe

2) SectionContext::tables

It was removed in apache/datafusion#9627

Many of the tests rely on this method. Does each test get updated to a new API or do we implement our own version to start and then update the tests until it's not necessary?

3) New Trait Methods

I just marked these as TODO to get the upgrade to compile. See 8cd4fd4

[Michael-J-Ward]

Current Status

The initial steps are completed and python -m pytest now passing in #662 (modulo 1, discussed below).

TODO

Besides the failing test

• Debug last failing test from porting BuiltinFunctions: ecff357
• Decide if re-implementing SessionContext::tables is what we want to do
• (maybe?) Remove allow(deprecated) from udf.rs
• (maybe?) Ensure full test coverage of ported functions.rs