feat: support using FTS as a filter in vector search

[wojiaodoubao]

Close #4927

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.00000% with 56 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/scanner.rs	80.00%	25 Missing and 27 partials ⚠️
rust/lance-index/src/scalar/inverted/index.rs	84.61%	1 Missing and 1 partial ⚠️
rust/lance/src/dataset/fragment.rs	0.00%	0 Missing and 1 partial ⚠️
rust/lance/src/dataset/write/delete.rs	0.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[wojiaodoubao]

Hi @BubbleCal , sorry to bother. Could you help review when you have time, thanks very much!

[BubbleCal]

This feature is cool, but I think we still need to figure out the API and semantic things:

this PR assumes if both FTS and vector query are provided, then one of them would be as the filter of the other one (depends on prefilter param), I'd think it would be better to always apply filter by filter() method, we can change this method to take an enum param which can be one of string, FTSQuery, VectorQuery.

[BubbleCal]

How would this new feature work if the query contains filter, FTS and vector query all?

[wojiaodoubao]

If prefilter=true, scanner will first perform the filter, then perform fts based on filter result, then perform vector search based on fts result.

If prefilter=false, scanner will first perform vector search, then filter the result with fts, then filter the result with filter.

[BubbleCal]

I'd expect if the FTS is a post-filter, then we should get vector search results (mean there should be '_distance' column), does it work like this?

[wojiaodoubao]

For all 3 cases in the unit tests, the result schema is (id, vector, text, _distance, _score). So yes it contains '_distance', but it shouldn't contain '_score' since fts is used as filter. This is related to the semantic and api issue above, let me fix them together.

[wojiaodoubao]

This feature is cool, but I think we still need to figure out the API and semantic things:
this PR assumes if both FTS and vector query are provided, then one of them would be as the filter of the other one (depends on prefilter param), I'd think it would be better to always apply filter by filter() method, we can change this method to take an enum param which can be one of string, FTSQuery, VectorQuery.

@BubbleCal Thanks for your nice suggestion! The current implementation does have some semantic confusion. I previously thought "using FTS as a filter for vector search" and "using vector search as a filter for FTS" were equivalent. Specifically, vector search with prefilter=true and FTS as filter is equivalent to full text search with prefilter=false and vector search as filter.

However, they are actually different because the returned schema is not the same. The results of an FTS query should include _score instead of _distance, while the results of a vector search should include _distance instead of _score.

I like your idea "apply filter by filter() method, we can change this method to take an enum param which can be one of string, FTSQuery, VectorQuery". This approach allows us to distinguish between filter and search. Additionally, I think the enum param can be slightly modified to support cases where both string & fts or string & vector are used as filters simultaneously.

enum FilterParam {
    Sql(String),
    Fts(FullTextSearchQuery),
    Vector(Query),
    SqlAndFts(String, FullTextSearchQuery),
    SqlAndVector(String, Query)
}

Let me fix it.

[BubbleCal]

@wjones127 I'm under the impression that we plan to make some changes to the query builder like:
ds.filter("x>0").filter("x<10").nearest(vec) is equal to ds.filter("x>0 and x<10").nearest(vec)

Could you confirm? If so I think it's a good time to implement this, and then we can have the filter enum simpler:

enum Filter {
    Sql(String),
    Fts(FullTextSearchQuery),
    Vector(Query),
}

cc @wojiaodoubao

[wojiaodoubao]

Hi @BubbleCal @wjones127 , please help review when you have time, thanks very much!

[wjones127]

@wjones127 I'm under the impression that we plan to make some changes to the query builder like: ds.filter("x>0").filter("x<10").nearest(vec) is equal to ds.filter("x>0 and x<10").nearest(vec)

Could you confirm?

Yeah I think we are getting to the place where we need to replace the Scanner implementation entirely. What we really need is a logical plan builder, so users can compose arbitrary plans together.

That's a larger project. I'm personally okay doing something basic here to support this, with the idea in mind that we will soon replace the scanner API entirely. But that's just my opinion, I don't know what others think.

[wojiaodoubao]

... we are getting to the place where we need to replace the Scanner implementation entirely. What we really need is a logical plan builder, so users can compose arbitrary plans together.

Hi @wjones127 , great to see that lance is going to work on the logical plan! I remembered there was an issue about this: #1782. I'd be happy to help if there is anything I can do.

Using fts/vector as a filter is useful, so I'd like to have a basic impl if it is possible.

[wojiaodoubao]

Hi @wjones127 @westonpace @BubbleCal , sorry to bother you. Could you help review this when you have time, thanks very much!

[BubbleCal]

Great work!
Let's create a follow-up issue for adding this to python API!