Use Solr for file listing on dataset page

[pdurbin]

In the FRD about improving the organization and display of files as mentioned on the Google Group we've been talking about the possibility of building the list of files on the dataset page using Solr rather than the database. Solr provides a better search experience than the database and we could make use of facets at the file level. I'll include a preliminary mockup below of how facets might look.

The primary challenge is how to handle files from old published versions of datasets because they are currently not indexed. A switch from the database to Solr should be carefully considered for a few reasons. We have already identified performance problems with indexing production data (#50) and indexing permissions (#2036). In addition, managing the lifecycle of Solr documents and their discoverability rules (#57) is already complex. We don't want to change the rules of when cards are shown on the homepage/dataverse page or in MyData.

[landreev]

These are my summarized thoughts on the issue (reposted from the dataverse list):

After talking offline to some developers about this, I figured I would bring this up one more time.

Namely, I'm talking about switching to using solr for viewing files. (Please read the whole thing before deciding that you hate my idea!)

So far, we have the following good reasons to proceed with this:

1. Duplication of code in the current setup. Search cards on the dataverse page do essentially the same as the file cells on the dataset page. With 2 very different processes of retrieving the data.

Counter arguments:

• it appears that we'll still have to maintain some code that renders file information retrieved from the database: the mydata page. maybe the code could be unified and shared between mydata and dataset pages instead?
• current setup has one big advantage - it's something that's working already more or less.

2. Pagination - something we would get "for free" with solr.

Counter argument:

According to Raman, pagination was fairly straightforward to implement on his mydata page.
In fact, it should be even simpler on the dataset page. Mydata may contain objects combined from different sources. The files on the dataset are just the filemetadatas from a single version, obtainable with a single query. You know right away how many you got. It really is easy to split these into several page chunks.

3. Searching for files/jumping between pages to specific files - straightforward with solr.

Counter argument:

This will be an issue, if and once we extend our metadata blocks to cover files.
As of now, the file metadata come from 2 db tables. Searching these directly in the db would be both straightforward and, likely, faster.

4. Facets. (pro and con same as above)

There is of course also 5., or 0. argument - the current dataset page is a big unmanageable mess that we've been meaning to reorganize and split.
But that is an argument in favor of splitting/reorganizing the page, not going to solr necessarily. We've been meaning to address this anyway, regardless of solr.

So, I'm not arguing for not going to solr. I am proposing to seriously consider not going to solr in 4.2. And, instead, to address that once we have extra metadata on the files level to deal with. Instead, let's set a more manageable, incremental 4.2 goal - separate the current page into view and edit (and upload) components; implement the new edit-on-demand scheme, instead of the current edit-all-files-at once; and add pagination (and, maybe, search) to the view page, while leaving the metadata retrieval as is.

Otherwise, I'm worried that fully reorganizing the page AND switching to solr is too ambitious a goal. That will turn 4.2 into another endless release. With, at best, even if we don't break anything in the process, nothing extra to show to the users for that extra effort.

[landreev]

(github messed up the formatting, but should still be readable)

[pdurbin]

I agree with everything @landreev has to say. (Oh, I fixed the formatting a bit, by the way.)

No matter what milestone this feature ends up in I think a user story for using Solr for file listing on dataset page will help us write an automated test to validate that the feature is working as desired. I just started a Google doc for this and would love feedback (public comments are enabled).

[pdurbin]

On Monday afternoon we met and decided that while we are all enthusiastic about the features of Solr we could make use of (especially faceting and nice search) we can't use Solr for files in the 4.2 timeframe because all the data necessary (past versions of published datasets) is not in Solr.

I started writing some tests to illustrate the problem of the data simply not being in Solr there based on scenarios in a Google Doc (more scenarios would be appreciated). My approach would be to keep adding to our test suite before making any changing to indexing. Then start changing the indexing rules (#528 in particular might change) to support this feature.

This is all to say that I'm moving this suggestion out of 4.2 milestone.