Design librarian process & UI for merging duplicate Subjects

[george08]

Can we please get rid of the duplicate variations on search results pages for subjects?

E.g.
http://openlibrary.org/search/subjects?q=New%20York

⚠️ EDIT: Administrative edit by @mekarpeles -- please jump to #65 (comment) to see recent context + proposal for implementing a solution to this issue.

[bencomp]

Duplicate variations are cool. That there are so many of them is a data problem, not a search problem per se. Different spellings may make a huge difference and so do Paris (place) vs. Paris (person).

A serious problem I face here is getting 0 results when I click the New York example. Somehow OL doesn't understand the space, neither kept as %20 nor translated to +. Only when I explicitly write new_york I get results.

[bencomp]

It appears my problem was solved by 06b0da2, thanks Anand!

I now realise the problem with duplicates/variations is a bit different than I thought. The search query for New York yields subjects like New York University. On page 1 are three listings for New York University:

• New York University (subject) 56 books
• New York university (subject) 55 books
• New York University (org) 47 books

The URLs for these results are the same, which means the number of books is wrong in at least two of these three results. The 'org' results are probably books in which the original records have the subject in a special (sub)field, but there is no way normal users can edit this in individual books.

I'm pretty sure this is a problem with how Solr gets updated.

[hornc]

In https://openlibrary.org/search/subjects?q=University+York the subject and org versions with the same URLs show the same counts...

but https://openlibrary.org/search/subjects?q=University+London
has
London University College 40 books, subject
London University College 39 books, org

and the same URLs, so this is still an issue.

SOLR updating issue?

[brad2014]

@hornc , could you please opine as to the appropriate priority of this, by setting a label?

[tfmorris]

George's query returns 0 results (problem solved? :) ) Likely related to #322

The general problem is that subjects need to be objects, not string, and have aliases, associated metadata, etc.

[hornc]

Yes, I think this can be closed. #322 is a more specific and current issue that can be dealt with independently. Any other current Subject related issues should be raised separately with current examples.

[tfmorris]

I think this is still a valid issue. Although George's original query is broken due to a different bug (#322), the problem she reported isn't fixed. The queries that @hornc posted here in 2017: #65 (comment) still demonstrate the problem.

The first three hits:

York University (Toronto, Ont.) 40 books, subject
York University (Toronto, Ont.). 24 books, org
York University (Toronto, Ont.). 24 books, subject

resolve to only 2 uniq URLs which differ only be a single trailing period. This is precisely the same as Ben's example from 2014 (except there the difference was a single letter case difference).

Additionally, the two subject URLs have work counts of 29 and 19, not 40 and 24, respectively, so the counts reported on the search results page is incorrect, but I'm guessing that if we fix the duplication, the counts will take care of themselves, so why don't we focus this issue on that? It's also the problem that George reported in 2011 and was confirmed in 2014 and again in 2017.

There are also additional search hits which should be merged:

Toronto York University 4 books, subject
York University (Toronto) 1 book, org
York University, Toronto 1 book, subject

but if we take care of the simple, common cases to start, we'll have a 90% solution.

On a more general note, I think favoring "current" (ie more modern) reports over historical historical ones is a bad idea in general because it loses the history of research that was done and obscures how long the problem has existed (8 years in this case).