Dataverse Page - 4.6.2 Regression #1 - Dataset Cards/Thumbnails

[raprasad]

Scenario:

• Local dataverse system running develop as of 5/24/2017
• Root dataverse page. System has:
  • 1 Dataverse (root)
  • 4 Datasets (these are heavily versioned)
  • 13 files

Each home page refresh runs 890 queries

• compared to 78 queries in 4.6.1

These 3 queries are run 200 times each

(1) COUNT: 200   QUERY(truncated): SELECT ID, UNF, ARCHIVENOTE, ARCHIVETIME, 
CREATETIME, DEACCESSIONLINK, INREVIEW, LASTUPDATETIME, MINORVERSIONNUMBER,
 RELEASETIME, VERSION, VERSIONNOTE, VERSIONNUMBER, VERSIONSTATE, DATASET_ID, termsOfUseAndAccess_id 
FROM DATASETVERSION WHERE (termsOfUseAndAccess_id = ?)
----------------------------------------
(2) COUNT: 200   QUERY(truncated): SELECT ID, CREATETIME, NAME, USAGECOUNT, 
DATAVERSE_ID, termsOfUseAndAccess_id
 FROM TEMPLATE WHERE (termsOfUseAndAccess_id = ?)
----------------------------------------
(3) COUNT: 200   QUERY(truncated): SELECT ID, AVAILABILITYSTATUS, CITATIONREQUIREMENTS,
 CONDITIONS, CONFIDENTIALITYDECLARATION, CONTACTFORACCESS, DATAACCESSPLACE, 
DEPOSITORREQUIREMENTS, DISCLAIMER, FILEACCESSREQUEST, LICENSE, ORIGINALARCHIVE, 
RESTRICTIONS, SIZEOFCOLLECTION, SPECIALPERMISSIONS, STUDYCOMPLETION, 
TERMSOFACCESS, TERMSOFUSE FROM TERMSOFUSEANDACCESS WHERE (ID = ?)

[raprasad]

@kcondon @pdurbin

This appears to have been introduced after 4.6.1

Scenario:

• Same data as described above
• Downloaded, compiled ran 4.6.1 against the same database

Each home page refresh runs between 78 and 101 queries*

• (* logged and not-logged in, respectively)

----------------------------------------
(1) COUNT: 20   QUERY(truncated): SELECT t0.ID, t0.DESCRIPTION, t0.DISPLAYNAME, t0.GROUPALIAS, t0.GROUPALIASINOWNER, t0.OWNER_ID FROM EXPLICITGROUP t0, ExplicitGroup_CONTAINEDROLEASSIGNEES t1 WHERE (((t0.OWNER_ID = ?) AND (t1.CONTAINEDROLEASSIGNEES = ?)) AND (t1.ExplicitGroup_ID = t0.ID))
----------------------------------------
(2) COUNT: 18   QUERY(truncated): SELECT NAME, CONTENT FROM SETTING WHERE (NAME = ?)
----------------------------------------
(3) COUNT: 13   QUERY(truncated): SELECT ID, DISPLAYNAME, NAME, owner_id FROM METADATABLOCK WHERE (ID = ?)
----------------------------------------
(4) COUNT: 10   QUERY(truncated): SELECT ID, ASSIGNEEIDENTIFIER, PRIVATEURLTOKEN, DEFINITIONPOINT_ID, ROLE_ID FROM ROLEASSIGNMENT WHERE ((ASSIGNEEIDENTIFIER = ?) AND (DEFINITIONPOINT_ID = ?))
----------------------------------------
(5) COUNT: 6   QUERY(truncated): SELECT ID, ADVANCEDSEARCHFIELDTYPE, ALLOWCONTROLLEDVOCABULARY, ALLOWMULTIPLES, description, DISPLAYFORMAT, DISPLAYONCREATE, DISPLAYORDER, FACETABLE, FIELDTYPE, name, REQUIRED, title, WATERMARK, METADATABLOCK_ID, PARENTDATASETFIELDTYPE_ID FROM DATASETFIELDTYPE WHERE (ID = ?)
----------------------------------------
(6) COUNT: 4   QUERY(truncated): SELECT t0.ID, t0.DESCRIPTION, t0.DISPLAYNAME, t0.GROUPALIAS, t0.GROUPALIASINOWNER, t0.OWNER_ID FROM EXPLICITGROUP t0, ExplicitGroup_CONTAINEDROLEASSIGNEES t1 WHERE ((t1.CONTAINEDROLEASSIGNEES = ?) AND (t1.ExplicitGroup_ID = t0.ID))
----------------------------------------
(7) COUNT: 2   QUERY(truncated): SELECT ID, ADVANCEDSEARCHFIELDTYPE, ALLOWCONTROLLEDVOCABULARY, ALLOWMULTIPLES, description, DISPLAYFORMAT, DISPLAYONCREATE, DISPLAYORDER, FACETABLE, FIELDTYPE, name, REQUIRED, title, WATERMARK, METADATABLOCK_ID, PARENTDATASETFIELDTYPE_ID FROM DATASETFIELDTYPE WHERE (name = ?)

[pdurbin]

@raprasad bleh! Thanks for checking if it's a regression or not.

[raprasad]

For the scenario in initial description, the cause is these 5 lines in the SearchServiceBean:

9ddd0a4#diff-b55512aefa9f8bcc2670df32e065f3dcR465

                DvObject dvObject = dvObjectService.findDvObject(entityid);
                if (dvObject != null) {
                    Dataset dataset = (Dataset) dvObject;
                    solrSearchResult.setDatasetThumbnail(dataset.getDatasetThumbnail());
                }

Those 5 lines eventually call the DatasetUtil file and the method getThumbnail(Dataset dataset)

---

The test page currently creates 922 total queries--the majority of which are:

• 39 queries: DvObject dvObject = dvObjectService.findDvObject(entityid);
• 781 queries: solrSearchResult.setDatasetThumbnail(dataset.getDatasetThumbnail());
  • All queries are from: dataset.getDatasetThumbnail() in the line above
  • next call: Dataset : getDatasetThumbnail()

problem area -- or maybe it shouldn't be called at all for Dataverse page rendering

• eventual culprit: -> DatasetUtil : getThumbnail(Dataset dataset)

[raprasad]

More info on DatasetUtil : getThumbnail(Dataset dataset)

My latest test page creates 922 queries.

This line is responsible for 437 of those queries:

String title = dataset.getLatestVersion().getTitle();

Breakdown:

• .getLatestVersion() - 383 queries
• .getTitle() - 54 queries

[raprasad]

This is best left for architecture meeting but there is a relatively easy solution to fix this for reads not writes. (credit to FB 7+ years ago) Specifically:

• Save 2nd version of dataset version metadata as JSON string.
  • acts as "cached" copy--saved each time you save the dataset with published versions being static
• Retrieve it in a single query as a Map
• Navigate to the needed attribute
• Had tested it on the miniverse in Nov/Dec.)

[raprasad]

Ayy!! I may split this into a second issue.

(Rest of info moved to #3858)

[pdurbin]

String title = dataset.getLatestVersion().getTitle();

@raprasad thanks for finding this. It's only used or debugging and should be a trivial change to replace it with the id of the dataset. This reminds me of #2110.