Index "File Tag" and "Access" (Public vs. Restricted) facets

[pdurbin]

Even if facets only show on the dataverse page and not the dataset page, we'd like to start indexing tags (formerly categories) at the file level so we can facet on them.

We could also index other fields at the file level, such as:

• More detailed file type ("RData" instead of just "tabular")
• If the file is restricted or not. See also add "is_restricted" to solr cards #2325
• ...

[pdurbin]

As I was discussing with @mheppler this data about file types is already indexed in Solr:

        "fileTypeDisplay":"Adobe PDF",
        "fileContentType":"application/pdf",
        "fileType":["Adobe PDF",
          "document"],
        "fileTypeGroupFacet":"document",

Currently we only show "document" for PDFs because we use the fileTypeGroupFacet which shows up as "File Type" in the GUI:

It would probably be nice to be able to facet on stuff like "Adobe PDF" (fileTypeDisplay) in the GUI. We'd need to give it a label like "Specific File Type" or something.

I'm less sure that we should add a facet for "application/pdf" (fileContentType). I guess I'd call this "File Content-Type" or "File MIME Type".

Update: It's a one line change (solrQuery.addFacetField(SearchFields.FILE_TYPE_FRIENDLY);) to add the "Adobe PDF" version but we need to figure out the labels for these. "File Type Friendly" sounds weird, obviously (I haven't committed this code). The one currently called "File Type" is really a grouping of file types into buckets like "tabulardata" or "image":

[pdurbin]

With around 8000 datasets from production data indexed locally and here's how the new "File Tags" facet is looking:

[pdurbin]

If the file is restricted or not. See also #2325

After discussing this with @landreev I'm going to defer this task for 4.2. It turns out to be more complicated that I expected per my comment at #2325 (comment)

Update: I think this might work but I need to check with someone datafileSolrInputDocument.addField(SearchFields.ACCESS, fileMetadata.isRestricted() ? "Restricted" : "Public");

It yield a mix of public and restricted at least:

This simple boolean check does not query the permissions system at all. This might be the wrong way to do the check, per my comment at #2325 (comment)

Here's how I identified some datasets with restricted files according to the datasetmetadata table: select distinct dataset_id,fm.restricted from filemetadata fm, datasetversion dsv where dsv.id=fm.datasetversion_id and fm.restricted=true limit 10;

[pdurbin]

@eaquigley as we discussed I'm giving this ticket to you to review. I've put v. 4.2 build 23-0178b43 on dataverse-internal.iq for you see the change.

This is what has changed:

• I added a "File Tags" facet and here's where you can see how it looks with ~8000 datasets indexed from production: Index "File Tag" and "Access" (Public vs. Restricted) facets #2466 (comment)

This is what I haven't changed but what I've commented on in this ticket:

• I did not add a "Access" facet (Public vs. Restricted) because per this comment I've been told by @landreev that it's not as simply as I hoped: Index "File Tag" and "Access" (Public vs. Restricted) facets #2466 (comment)
• I did not add a second version of "File Type" that would show us PDF rather than "document" because I'm not sure if we want it or what we'd call it. The data is in Solr already (and has been since 4.0) so it would be easy to make a change in this area if we'd like to. Please see Index "File Tag" and "Access" (Public vs. Restricted) facets #2466 (comment)

The only other thing I can think to facet on at the file-level is size but until we figure out the UI for ranges (I like the design for date ranges at #70 (comment) ) and have time to dedicate to it, I don't think it's worth spending time on right now. Definitely not for 4.2.

[eaquigley]

@pdurbin one note, can you change File Tags to be File Tag. All other facets are singular.

Regarding the Access facet, if you have time this week, could you look into this and determine how risky it is to do this and the technical difficulty?