"engineering librarian" staff search results are not very relevant or similar to libguides results

[sandbergja]

Expected behavior

When I search https://allsearch-api.princeton.edu/search/staff?query=engineering%20librarian, it should return Willow Dressel, Kelee Pacion, and Zac Painter

Actual behavior

It returns other librarians who don't work as much with engineering

Impact of this bug

Unhelpful results for the user

[maxkadel]

I've investigated this some.

Right now, if you use the query on the model and search for engineering librarian, Willow and Zac are in the results, but they're not sorted to the top (they're 4th and 6th, respectively).

Kelee does not appear in the results - I think this may be because the term "librarian" does not appear in the fields that are indexed for the searchable field, whereas it does appear for all of the records that are returned.

Recreating

bundle exec rake servers:start
bundle exec rails c

LibraryStaffLoadingService.new.run
results = LibraryStaffRecord.query('engineering librarian')
results.map { |person| [person.name, person.searchable] }
kelee = LibraryStaffRecord.query('pacion')
kelee.map { |person| [person.name, person.searchable] }

PostgreSQL documentation on text search

[maxkadel]

For Kelee - could add "bio" to the searchable, and she would be more likely to be a result for this search.

[maxkadel]

Query that mostly worked!

Part of the issue is that we need to match the dictionary to how the field was ingested.

select name, ts_rank(searchable, websearch_to_tsquery('unaccented_dict', unaccent('engineering librarian'))) + ts_rank(name_searchable, websearch_to_tsquery('unaccented_simple_dict', unaccent('engineering librarian'))) AS rank from library_staff_records order by rank desc;

[maxkadel]

Kelee Pacion is in the database search results; however, she's still not in the first three results.

The best solution may be to add "librarian" to somewhere else in her record, rather than continue to try to futz with the search or indexing.

[kevinreiss]

@beth-german @tm3211 could have a short discussion about how much effort we should invest in programming to make this better vs. some curation of the data we have in airtable per Max's last comment?