Match Phrase Prefix

[Venommmm]

Is it possible to match query like 'build softwa" for index item "build software"? If there are items in index simular to 'build xxxx", query 'build softwa" may be wont be top result (using match query).

[mschoch]

Match Phrase Prefix search is possible, but it's currently not implemented.

There are two ways I can think of implementing it (there may be more).

The first is simple, but won't perform as well, that is to first expand the prefix. So "softwa" would expand to a few terms, one of which is "software". Then you combine that with the other phrase terms, and do N separate phrase searches.

The second solution is more complicated, but probably performs better, particularly in cases where the prefix matches a lot of terms. You would first find documents that contain the start of the phrase, and then you would combine that with a query that expanded the prefix and also ensured the position offsets were 1. Doing it this way is a bit more complicated but starting with a search on phrase probably cuts down on the candidate documents considerably in practice.

If you're interested in work on this I would suggest starting with the first approach. It's relatively straightforward enhancement to get started with developing bleve. Let us know if you try this and have questions.

[Venommmm]

Here is my noob code (using this with twitter typeahead), seems it works nicely.
```
var searchResults *bleve.SearchResult
words := strings.Fields(q)
if len(words) == 1 {

    query_list := make([]bleve.Query, 2)
    query_list[0] = bleve.NewPrefixQuery(q)
    query_list[1] = bleve.NewMatchQuery(q)
    query := bleve.NewDisjunctionQuery(query_list)

    search := bleve.NewSearchRequestOptions(query, 1000, 0, false)
    searchResults, _ = Search_index[Folder].Search(search)

} else { // more than 1 word
    phrase := ""
    k := 0
    for k != len(words)-1 {
        phrase += words[k] + " "
        k++
    }

    phrase = phrase[0 : len(phrase)-1]
    prefix := words[len(words)-1]

    // query list
    query_list := make([]bleve.Query, 2)

    query1 := bleve.NewMatchPhraseQuery(phrase)

    query_list[0] = query1
    query_list[1] = bleve.NewPrefixQuery(prefix)

    query := bleve.NewConjunctionQuery(query_list)

    search := bleve.NewSearchRequestOptions(query, 5, 0, false)
    searchResults, _ = Search_index[Folder].Search(search)

    // if no results, try match prefix as match
    if len(searchResults.Hits) == 0 {
        query_list[0] = query1
        query_list[1] = bleve.NewMatchQuery(prefix)
        search := bleve.NewSearchRequestOptions(query, 5, 0, false)
        searchResults, _ = Search_index[Folder].Search(search)
    }

}

[Venommmm]

Additionally, i am using framework revel and when i start my app (i am using like 20 different indexes) i get message watcher.go:124: Failed to watch, no space left on device. This can be ignored or it can affect bleve?

[mschoch]

Yeah, I think you're emulating at the user level pretty closely. Implementing it inside bleve would be similar but using index searchers, and making one pass instead of multiple.

One thing I strongly advise against is ignoring the error value being returned. In the best case this leads to confusion when things don't work, and an important message was being ignored, and in the worse case it can lead to panics, because you accidentally continue operating on structures that are nil.

And regarding no space left on device I don't think that can ever really be ignored. If the disk is full bleve won't be able to index data, and even closing/reopening the index could fail.

[Venommmm]

Thank you, for your answer and great code. Can you add some information to documentation about non-included language packs in blevex. I think this may be helpful for other users.

[Venommmm]

I found a problem with code: a query like "xxxx stop-word-like-preposition" will give zero results if used with text analyzer which removes stop-words. Matching only "xxxx" in this situation may not correct with typeahead. On the other hand analyzer with removing stop-words is nice.
It can be fixed by matching only "xxxx" and then try to sort with levenshtein function.

[mschoch]

Regarding additional documentation, can you create a separate bug and describe what you want. I try to use these issues as tasks, so it helps if they focus on just one thing at a time.

Regarding the stop-word issue that is expected behavior. If during analysis you choose to remove stop words, then you have to use analysis at search time too. This means using MatchPhrase instead of Phrase.

I'm not sure what you mean by "try to sort with levenshtein function".

[mschoch]

Scheduling this for 1.X since new searches would be additive additions to the API and not require changes.

[tmm1]

@Venommmm thanks for your example. Here's what I did:

var query bleve.Query
words := strings.Fields(q)
if len(words) == 1 {
    query = bleve.NewDisjunctionQuery([]bleve.Query{
        bleve.NewPrefixQuery(q),
        bleve.NewMatchQuery(q),
    })
} else {
    var phrase string
    if len(words) == 2 {
        phrase = words[0]
    } else {
        phrase = strings.Join(words[0:len(words)-2], " ")
    }

    query = bleve.NewDisjunctionQuery([]bleve.Query{
        bleve.NewMatchPhraseQuery(q), // whole thing as a phrase, or..
        bleve.NewConjunctionQuery([]bleve.Query{ // phrase + prefix
            bleve.NewMatchPhraseQuery(phrase),
            bleve.NewPrefixQuery(words[len(words)-1]),
        }),
    })
}

Still doesn't really handle analysis correctly though..

I would love to see bleve offer a native MatchPhrasePrefixQuery. I can help start an implementation if no one has already. Not very familiar with bleve internals so any pointers would be appreciated.

[tmm1]

My implementation fails when searching with stop words.. for example searching for "how to" will not return matching results.

[mschoch]

@tmm1 what behavior are you trying to get? Since "how" and "to" are both stop words, the only way you could possibly get search results would be to not treat them as stop words and put them into the index in the first place.

[tmm1]

That's probably what I need to do. How can I whitelist stop words?