Skip to content

5207 dataverse facet #5697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kcondon merged 12 commits into from
Aug 9, 2019
Merged

5207 dataverse facet #5697

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b2ab9ad
Facet Category and Labels
JayanthyChengan Mar 28, 2019
4e15e4e
merge with develop
JayanthyChengan Mar 28, 2019
9625738
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Mar 28, 2019
261c6da
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Apr 9, 2019
b7bacc9
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Apr 25, 2019
fdf59e3
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Jul 12, 2019
0f438ae
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Jul 19, 2019
863d883
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Jul 24, 2019
7d9137f
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Jul 26, 2019
8cb0040
Added try/catch to handle missing property in bundle ( for user enter…
JayanthyChengan Jul 30, 2019
b7d0c17
Merge remote-tracking branch 'remotes/origin/develop' into 5207-datav…
JayanthyChengan Aug 6, 2019
fbae91b
Removed the staticSearchFields that are not facets from bundle.proper…
JayanthyChengan Aug 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import edu.harvard.iq.dataverse.dataset.DatasetThumbnail;
import edu.harvard.iq.dataverse.dataset.DatasetUtil;
import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
import edu.harvard.iq.dataverse.util.BundleUtil;
import edu.harvard.iq.dataverse.util.FileUtil;
import edu.harvard.iq.dataverse.util.SystemConfig;
import java.io.File;
Expand Down Expand Up @@ -428,9 +429,9 @@ public void search(boolean onlyDataRelatedToMe) {
}

// populate preview counts: https://redmine.hmdc.harvard.edu/issues/3560
previewCountbyType.put("dataverses", 0L);
previewCountbyType.put("datasets", 0L);
previewCountbyType.put("files", 0L);
previewCountbyType.put(BundleUtil.getStringFromBundle("dataverses"), 0L);
Copy link
Contributor

@scolapasta scolapasta Apr 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This text isn't actually shown anywhere, correct? I don't think there's a need to localize it.

Copy link
Contributor Author

@JayanthyChengan JayanthyChengan Apr 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scolapasta - This shows in the facets

image

Copy link
Contributor

@scolapasta scolapasta Apr 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JayanthyChengan Sorry, what I meant to say is that the text isn't gotten from this previewCountbyType hashmap, as far as I can tell (in the xhtml, I see calls to "{bundle['dataverse.results.types.dataverses']}" and "{bundle['dataverse.results.types.datasets']}". If that's true, then we don't need to localize the internal keys to hashmaps like this.

(or am I missing something?)

Copy link
Contributor Author

@JayanthyChengan JayanthyChengan Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scolapasta - I replied for this comment to your email. Please let me know, if I need to forward the email again. Thanks

Copy link
Contributor

@scolapasta scolapasta Jul 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JayanthyChengan Can you go ahead and forward the e-mail again? I couldn't find the reference to this specific comment.

Copy link
Contributor Author

@JayanthyChengan JayanthyChengan Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scolapasta - just forwarded the e-mail.

previewCountbyType.put(BundleUtil.getStringFromBundle("datasets"), 0L);
previewCountbyType.put(BundleUtil.getStringFromBundle("files"), 0L);
if (solrQueryResponseAllTypes != null) {
for (FacetCategory facetCategory : solrQueryResponseAllTypes.getTypeFacetCategories()) {
for (FacetLabel facetLabel : facetCategory.getFacetLabel()) {
Expand Down Expand Up @@ -765,15 +766,15 @@ public void setTypeFilterQuery(String typeFilterQuery) {
}

public Long getFacetCountDatasets() {
return findFacetCountByType("datasets");
return findFacetCountByType(BundleUtil.getStringFromBundle("datasets"));
}

public Long getFacetCountDataverses() {
return findFacetCountByType("dataverses");
return findFacetCountByType(BundleUtil.getStringFromBundle("dataverses"));
}

public Long getFacetCountFiles() {
return findFacetCountByType("files");
return findFacetCountByType(BundleUtil.getStringFromBundle("files"));
}

public String getSearchFieldRelevance() {
Expand Down
118 changes: 89 additions & 29 deletions src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import java.util.ListIterator;
import java.util.Map;
import java.util.Set;
import java.util.MissingResourceException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.ejb.EJB;
Expand Down Expand Up @@ -98,7 +99,7 @@ public class SearchServiceBean {
public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Dataverse> dataverses, String query, List<String> filterQueries, String sortField, String sortOrder, int paginationStart, boolean onlyDatatRelatedToMe, int numResultsPerPage) throws SearchException {
return search(dataverseRequest, dataverses, query, filterQueries, sortField, sortOrder, paginationStart, onlyDatatRelatedToMe, numResultsPerPage, true);
}

/**
* Import note: "onlyDatatRelatedToMe" relies on filterQueries for providing
* access to Private Data for the correct user
Expand All @@ -116,7 +117,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
* @param paginationStart
* @param onlyDatatRelatedToMe
* @param numResultsPerPage
* @param retrieveEntities - look up dvobject entities with .find() (potentially expensive!)
* @param retrieveEntities - look up dvobject entities with .find() (potentially expensive!)
* @return
* @throws SearchException
*/
Expand Down Expand Up @@ -170,7 +171,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
/**
* @todo: show highlight on file card?
* https://redmine.hmdc.harvard.edu/issues/3848
*/
*/
solrFieldsToHightlightOnMap.put(SearchFields.FILENAME_WITHOUT_EXTENSION, "Filename Without Extension");
solrFieldsToHightlightOnMap.put(SearchFields.FILE_TAG_SEARCHABLE, "File Tag");
List<DatasetFieldType> datasetFields = datasetFieldService.findAllOrderedById();
Expand Down Expand Up @@ -221,7 +222,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
* if advancedSearchField is true or false
*
*/

//I'm not sure if just adding null here is good for hte permissions system... i think it needs something
if(dataverses != null) {
for(Dataverse dataverse : dataverses) {
Expand All @@ -245,7 +246,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
solrQuery.addFilterQuery(permissionFilterQuery);
}
}

solrQuery.addFacetField(SearchFields.FILE_TYPE);
/**
* @todo: hide the extra line this shows in the GUI... at least it's
Expand Down Expand Up @@ -293,7 +294,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
solrQuery.setRows(numResultsPerPage);
logger.fine("Solr query:" + solrQuery);

// -----------------------------------
// -----------------------------------
// Make the solr query
// -----------------------------------
QueryResponse queryResponse = null;
Expand Down Expand Up @@ -441,12 +442,12 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
solrSearchResult.setDescriptionNoSnippet(description);
solrSearchResult.setDeaccessionReason(deaccessionReason);
solrSearchResult.setDvTree(dvTree);

String originSource = (String) solrDocument.getFieldValue(SearchFields.METADATA_SOURCE);
if (IndexServiceBean.HARVESTED.equals(originSource)) {
solrSearchResult.setHarvested(true);
}

/**
* @todo start using SearchConstants class here
*/
Expand All @@ -455,7 +456,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
solrSearchResult.setHtmlUrl(baseUrl + SystemConfig.DATAVERSE_PATH + identifier);
// Do not set the ImageUrl, let the search include fragment fill in
// the thumbnail, similarly to how the dataset and datafile cards
// are handled.
// are handled.
//solrSearchResult.setImageUrl(baseUrl + "/api/access/dvCardImage/" + entityid);
/**
* @todo Expose this API URL after "dvs" is changed to
Expand All @@ -468,11 +469,11 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
solrSearchResult.setApiUrl(baseUrl + "/api/datasets/" + entityid);
//Image url now set via thumbnail api
//solrSearchResult.setImageUrl(baseUrl + "/api/access/dsCardImage/" + datasetVersionId);
// No, we don't want to set the base64 thumbnails here.
// We want to do it inside SearchIncludeFragment, AND ONLY once the rest of the
// No, we don't want to set the base64 thumbnails here.
// We want to do it inside SearchIncludeFragment, AND ONLY once the rest of the
// page has already loaded.
//DatasetVersion datasetVersion = datasetVersionService.find(datasetVersionId);
//if (datasetVersion != null){
//if (datasetVersion != null){
// solrSearchResult.setDatasetThumbnail(datasetVersion.getDataset().getDatasetThumbnail(datasetVersion));
//}
/**
Expand All @@ -489,10 +490,10 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers

solrSearchResult.setCitation(citation);
solrSearchResult.setCitationHtml(citationPlainHtml);

solrSearchResult.setIdentifierOfDataverse(identifierOfDataverse);
solrSearchResult.setNameOfDataverse(nameOfDataverse);

if (title != null) {
// solrSearchResult.setTitle((String) titles.get(0));
solrSearchResult.setTitle(title);
Expand Down Expand Up @@ -586,22 +587,49 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
FacetCategory facetCategory = new FacetCategory();
List<FacetLabel> facetLabelList = new ArrayList<>();
int numMetadataSources = 0;
String metadataBlockName = "";
String datasetFieldName = "";
/**
* To find the metadata block name to which the facetField belongs to
* ===facetField: authorName_ss metadatablockname : citation
* ===facetField: dvCategory metadatablockname : ""
*/
for (DatasetFieldType datasetField : datasetFields) {
String solrFieldNameForDataset = datasetField.getSolrField().getNameFacetable();
if (solrFieldNameForDataset != null && facetField.getName().equals(solrFieldNameForDataset)) {
metadataBlockName = datasetField.getMetadataBlock().getName() ;
datasetFieldName = datasetField.getName();
break;
}
}


for (FacetField.Count facetFieldCount : facetField.getValues()) {
/**
* @todo we do want to show the count for each facet
*/
// logger.info("field: " + facetField.getName() + " " + facetFieldCount.getName() + " (" + facetFieldCount.getCount() + ")");
String localefriendlyName = null;
if (facetFieldCount.getCount() > 0) {
FacetLabel facetLabel = new FacetLabel(facetFieldCount.getName(), facetFieldCount.getCount());
if(metadataBlockName.length() > 0 ) {
localefriendlyName = getLocaleTitle(datasetFieldName,facetFieldCount.getName(), metadataBlockName);
} else {
try {
localefriendlyName = BundleUtil.getStringFromPropertyFile(facetFieldCount.getName(), "Bundle");
} catch (Exception e) {
localefriendlyName = facetFieldCount.getName();
}
}
FacetLabel facetLabel = new FacetLabel(localefriendlyName, facetFieldCount.getCount());
// quote field facets
facetLabel.setFilterQuery(facetField.getName() + ":\"" + facetFieldCount.getName() + "\"");
facetLabelList.add(facetLabel);
if (facetField.getName().equals(SearchFields.PUBLICATION_STATUS)) {
if (facetLabel.getName().equals(IndexServiceBean.getUNPUBLISHED_STRING())) {
if (facetFieldCount.getName().equals(IndexServiceBean.getUNPUBLISHED_STRING())) {
unpublishedAvailable = true;
} else if (facetLabel.getName().equals(IndexServiceBean.getDRAFT_STRING())) {
} else if (facetFieldCount.getName().equals(IndexServiceBean.getDRAFT_STRING())) {
draftsAvailable = true;
} else if (facetLabel.getName().equals(IndexServiceBean.getDEACCESSIONED_STRING())) {
} else if (facetFieldCount.getName().equals(IndexServiceBean.getDEACCESSIONED_STRING())) {
deaccessionedAvailable = true;
}
}
Expand Down Expand Up @@ -652,14 +680,20 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
Logger.getLogger(SearchServiceBean.class.getName()).log(Level.SEVERE, null, ex);
}
if (staticSearchField != null && facetField.getName().equals(staticSearchField)) {
String[] parts = name.split("_");
StringBuilder stringBuilder = new StringBuilder();
for (String part : parts) {
stringBuilder.append(getCapitalizedName(part.toLowerCase()) + " ");
String friendlyName = BundleUtil.getStringFromBundle("staticSearchFields."+staticSearchField);
if(friendlyName != null && friendlyName.length() > 0) {
facetCategory.setFriendlyName(friendlyName);
} else {
String[] parts = name.split("_");
StringBuilder stringBuilder = new StringBuilder();
for (String part : parts) {
stringBuilder.append(getCapitalizedName(part.toLowerCase()) + " ");
}
String friendlyNameWithTrailingSpace = stringBuilder.toString();
friendlyName = friendlyNameWithTrailingSpace.replaceAll(" $", "");
facetCategory.setFriendlyName(friendlyName);
}
String friendlyNameWithTrailingSpace = stringBuilder.toString();
String friendlyName = friendlyNameWithTrailingSpace.replaceAll(" $", "");
facetCategory.setFriendlyName(friendlyName);

// logger.info("adding <<<" + staticSearchField + ":" + friendlyName + ">>>");
staticSolrFieldFriendlyNamesBySolrField.put(staticSearchField, friendlyName);
// stop examining the declared/static fields in the SearchFields object. we found a match
Expand Down Expand Up @@ -747,6 +781,31 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, List<Datavers
return solrQueryResponse;
}

public String getLocaleTitle(String title, String controlledvoc , String propertyfile) {

String output = "";
try {
if(controlledvoc != "" ) {
output = BundleUtil.getStringFromPropertyFile("controlledvocabulary." + title +"."+ controlledvoc.toLowerCase().replace(" ","_") , propertyfile);
} else {
output = BundleUtil.getStringFromPropertyFile("datasetfieldtype." + title + ".title", propertyfile);
}
} catch (MissingResourceException e1) {
if(controlledvoc != "" ) {
return controlledvoc;
} else {
return title;
}
}

if(output != null && output.length() >0) {
return output;
}
else
return title;
}


public String getCapitalizedName(String name) {
return Character.toUpperCase(name.charAt(0)) + name.substring(1);
}
Expand Down Expand Up @@ -782,7 +841,7 @@ private String getPermissionFilterQuery(DataverseRequest dataverseRequest, SolrQ
}

// ----------------------------------------------------
// (1) Is this a GuestUser?
// (1) Is this a GuestUser?
// Yes, see if GuestUser is part of any groups such as IP Groups.
// ----------------------------------------------------
if (user instanceof GuestUser) {
Expand Down Expand Up @@ -820,9 +879,9 @@ private String getPermissionFilterQuery(DataverseRequest dataverseRequest, SolrQ
solrQuery.addFacetField(SearchFields.PUBLICATION_STATUS);

// ----------------------------------------------------
// (3) Is this a Super User?
// (3) Is this a Super User?
// Yes, give back everything
// ----------------------------------------------------
// ----------------------------------------------------
if (au.isSuperuser()) {
// dangerous because this user will be able to see
// EVERYTHING in Solr with no regard to permissions!
Expand All @@ -834,7 +893,7 @@ private String getPermissionFilterQuery(DataverseRequest dataverseRequest, SolrQ
// (4) User is logged in AND onlyDatatRelatedToMe == true
// Yes, give back everything -> the settings will be in
// the filterqueries given to search
// ----------------------------------------------------
// ----------------------------------------------------
if (onlyDatatRelatedToMe == true) {
if (systemConfig.myDataDoesNotUsePermissionDocs()) {
logger.fine("old 4.2 behavior: MyData is not using Solr permission docs");
Expand Down Expand Up @@ -907,3 +966,4 @@ private String getPermissionFilterQuery(DataverseRequest dataverseRequest, SolrQ
}

}

2 changes: 1 addition & 1 deletion src/main/java/edu/harvard/iq/dataverse/util/BundleUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ public static String getStringFromBundle(String key) {
}

public static String getStringFromBundle(String key, List<String> arguments) {
ResourceBundle bundle = getResourceBundle(defaultBundleFile , null);
ResourceBundle bundle = getResourceBundle(defaultBundleFile );
if (bundle == null) {
return null;
}
Expand Down
41 changes: 41 additions & 0 deletions src/main/java/propertyFiles/Bundle.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2357,3 +2357,44 @@ rtabfileparser.ioexception.parser2=R Tab File Parser: varQnty=0 in the dataset m

#ConfigureFragmentBean.java
configurefragmentbean.apiTokenGenerated=API Token will be generated. Please keep it secure as you would do with a password.

#FacetCategory - staticSearchFields
staticSearchFields.dvCategory=Dataverse Category
staticSearchFields.metadataSource=Metadata Source
staticSearchFields.publicationDate=Publication Year
staticSearchFields.fileTypeGroupFacet=File Type
staticSearchFields.dvObjectType=Type
staticSearchFields.fileTag=File Tag
staticSearchFields.fileAccess=Access
staticSearchFields.publicationStatus=Publication Status

#dataverse category - Facet Labels
Researcher=Researcher
Research\u0020Project=Research Project
Journal=Journal
Organization\u0020or\u0020Institution=Organization or Institution
Teaching\u0020Course=Teaching Course
Research\u0020Group=Research Group
Laboratory=Laboratory
Department=Department
Uncategorized=Uncategorized

#filetype - Facet Labels
Document=Document
Text=Text
Tabular\u0020Data=Tabular Data
Data=Data
FITS=FITS
Shape=Shape
Image=Image
Network\u0020Data=Network Data
Unknown=Unknown
Documentation=Documentation
Code=Code
Archive=Archive
Audio=Audio
Video=Video

#access - Facet Labels
Public=Public
Restricted=Restricted