Expose DataFusion extensions (plan nodes, plan optimizer rules)

[wjones127]

The code in dataset/scanner.rs has gotten extremely complicated, to a point where it is hard to test. Before we make any improvements, we need to refactor this to be easier to test and extend.

In addition, outside codebases may wish to extend Lance's capabilities by modifying or composing plans. For example, in LanceDB, we'll want to add a separate WAL that needs to be queried during KNN queries and scans.

Tasks

• Create a LogicalPlan for KNN search
• Write an extension for LogicalPlanBuilder and rewrite Scanner::create_plan() in terms of that.
• Create a custom PhysicalPlanner to convert out logical plans into relevant nodes
• Implement a PhysicalOptimizerRule for each of the optimizations we make:
  • Using vector indices (ANN)
  • Using scalar indices (ANN and scan)
• Make sure schema returned from scanner comes from the logical plan.
• metadata is returned sometimes but not all times. Make this behavior more consistent.
• make _rowid and _distance less awkward to deal with -- so that select([]).with_row_id() is just select ("_rowid") We should reserve that column name.
• don't return _distance unless select * or select _distance

[jayzhan211]

I'm interesting in improve datafusion extensibility for lance

[wjones127]

I think a good approach to this would be starting to design some logical plan nodes implementing UserDefinedLogicalNode.

From there, we can create a create_plan_v2() method that creates a logical plan instead of the ExecutionPlan. Later then we can physical planner and other things. This would keep the existing planning intact until we are ready to switch things over.

I'm interesting in improve datafusion extensibility for lance

Is there a particular goal you have in mind that you want to work towards?

[jayzhan211]

Is there a particular goal you have in mind that you want to work towards?

None yet. I would like to know how lance built on datafusion and what areas could be improved.

I think a good approach to this would be starting to design some logical plan nodes implementing UserDefinedLogicalNode.

Probably I could start from this! Create a LogicalPlan for KNN search

[wjones127]

@jayzhan211 If you want a smaller issue to get started with, this might be a better one: #1927 It will also have a more immediate pay off.