Subject search breaks when keyword "new" present

[bfalling]

e.g. http://openlibrary.org/search/subjects?q=New%20York

[hornc]

This might be an issue specific to 'New York' , other searches with spaces do give results, e.g. "North York", "West Africa", "vacuum tubes" (for something non-geographical).

Strangely every book I can find about New York has an 'NY' or 'N.Y.' as part of subject, as if it has been standardised. The canonical New York subject seems to be https://openlibrary.org/subjects/place:ny

Is there a standard the prefers 'NY' to 'New York'? It does seem odd that the entire OL db has every reference to New York stored as NY.

[jessamynwest]

@hornc @bfalling Interesting observation. I see it with some two word searches and not for all.

New Mexico fails
New Jersey fails
New Delhi fails
(trying to find a two word phrase starting with NEW that doesn't fail, maybe it's a stop word?)
Atlantic City succeeds
George Clinton succeeds
Bill Clinton succeeds

If there was some standardization done around the state names, itwas before my time.

[LeadSongDog]

Subject searching on "New" as a single word finds zero hits: https://openlibrary.org/search/subjects?q=New
Title searching for the same single word gets 116,292 hits: https://openlibrary.org/search?q=New
Title search for "New York" finds 36,163: https://openlibrary.org/search?q=New+York
It seems plausible that at some point this search bug was worked around by changing the subject records to NY instead. An alternative is to identify it as a place when applicable: https://openlibrary.org/subjects/place:new_york
Now then, looking at (for instance) https://openlibrary.org/works/OL16856275W/New_York_University we see that the user sees "Published 1911 in New York", but that place is linked as https://openlibrary.org/search/subjects?q=New York rather than https://openlibrary.org/subjects/place:new_york
This seems like it should be an easy thing to fix, probably at

openlibrary/openlibrary/templates/books/edition-show.html

Line 87 in acb9d2d

<a href="/search/subjects?q=$p.replace('&','%26')" title="$_('Search for subjects about') $p">$p</a>$cond(loop.last, "", ", ")

if I'm not mistaken.

[hornc]

@jessamynwest, following up on the 'New' subject, I found a book that has 'New Order (Group)' as a subject https://openlibrary.org/works/OL4311941W/24_Hour_Party_People . It doesn't show up in a subject search. Using the format of the New Order subject page: https://openlibrary.org/subjects/new_order_(group) , I found the elusive 'New York' subject page: https://openlibrary.org/subjects/new_york

It exists, and there are 106 works that have 'New York' as a subject, and there are numerous other New York related subjects. They just don't show up on a subject search.

I'll take a look at the code to see if there is anything that treats subjects searches with 'New' in them differently.

[bfalling]

Could "new" be considered a stopword in SOLR? And if so, could this be related?

[jessamynwest]

That is my presumption, or a reserved word.

[mekarpeles]

#65