feat: support sql api for dataset

[yanghua]

Closes #3992

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 89.43662% with 15 lines in your changes missing coverage. Please review.

Project coverage is 80.25%. Comparing base (eb5aafe) to head (f940194).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/sql.rs	91.24%	7 Missing and 5 partials ⚠️
rust/lance/src/dataset.rs	40.00%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4086      +/-   ##
==========================================
+ Coverage   80.12%   80.25%   +0.12%     
==========================================
  Files         294      297       +3     
  Lines      103595   105211    +1616     
  Branches   103595   105211    +1616     
==========================================
+ Hits        83008    84432    +1424     
- Misses      17546    17706     +160     
- Partials     3041     3073      +32     

Flag	Coverage Δ
unittests	80.25% <89.43%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[westonpace]

I agree with @jackye1995 that SQL tends to be more of a database/catalog-level operation and not a table operation.

However, as a convenience (to enable things like sorting and aggregation) it's probably ok as a sort of utility API.

[yanghua]

The background of this requirement is that we need to let users run SQLs on the lightweight sample lance datasets.

The current way lacks flexibility. We use lance scanner to filter an arrow dataset, then forward it to DuckDB to run a SQL on it.

[Jay-ju]

@yanghua I have a question. Can the SQL mode here support fts and fast_search?

[Jay-ju]

@yanghua @jackye1995 Can this be updated? I want to do something based on this PR.

[yanghua]

Will refactor the PR, a little busy, sorry.

[jackye1995]

@Jay-ju can't you just use the DataFusion integration directly? https://lancedb.github.io/lance/integrations/datafusion/

[yanghua]

@jackye1995 please take a look when you have time.

[yanghua]

@jackye1995 Addressed your comments, please take a look.

[jackye1995]

looks good to me! just 1 minor comment for the mod