Versions API improvements (9763)

[landreev]

What this PR does / why we need it:

(copy-and-pasting from the comments in the linked issue):

I'm addressing the performance issues in the versions api via a combination of the following approaches:

• (optional) pagination is being added to /api/datasets/{id}/versions
• a new flag includeFiles is added to both /api/datasets/{id}/versions and /api/datasets/{id}/versions/{vid} (true by default), providing an option to drop the file information from the output
• when files are requested to be included, the filemetadatas are looked up using the "left joint fetch" optimizations that retrieve the information from the extended table tree in a single query, instead of allowing EJB to perform 1 SELECT query on each 1:1 relation for each FileMetadata entry. On a version with N filemetadatas this saves 3*N individual queries. This is not free resources-wise by any means (costs more memory primarily), but still appears to show measurable improvement on datasets with large numbers of versions and/or files.

The following real life datasets from IQSS prod. service are used in the sample tests below:

• 5255036: 237 versions (!), ~2,400 files. Files are relatively sparsely spread between versions.
• 4554342: 2 versions, 1 pub. 1 draft. ~25,000 files (all appear in both versions).
• 2710927: a control "reasonable" dataset, 100 files, 3 published versions.

/api/datasets/{id}/versions:

dataset id	# of versions	# of files	response size	v5.14 response time	new response time	new size w/out files	new time w/out files
5255036	237	2400	38MB	1m30s	48s	2MB	12s
4554342	2	25000	17MB	2m16s	12s	2.5K	1s
2710927	3	100	.2MB	1s	<1s	26K	<1s

/api/datasets/{id}/versions/{vid}:

dataset id	version	# of files in version	response size	v5.14 response time	new response time	new size w/out files	new time w/out files
5255036	87.1	12	24K	1s	<1s	12K	<1s
4554342	draft	25000	17MB	2m17s	12s	4K	<1s
2710927	3.0	100	78K	1s	<1s	9K	<1s

Note: /api/datasets/4554342/versions and /api/datasets/4554342/versions/draft produce the same amount of output because the former api was called without authentication, so only one, published version was included.

Note: tests above were run on the dedicated IQSS test system (not in prod., which is a beefier and faster system).

Which issue(s) this PR closes:

Closes #9763

Special notes for your reviewer:

Please take an extra look at the framework for caching the adjusted "embargo publication date" that I added, replacing the real time lookup that used to be in Dataset.getCitationDatet(), and especially the flyway script added in the PR.

Suggestions on how to test this:

The description above outlines the way I tested the branch during dev.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[pdurbin]

Just some quick comments.

[pdurbin]

Yes, good to remove commented out code.

[pdurbin]

I recognize this from #9693. Does it matter which PR is merged first?

[landreev]

I would prefer #9693 to be merged first, and then resolve the conflict in this branch (I added a database lookup optimization on top of the code introduced in that PR).

[pdurbin]

As discussed at standup, @landreev sounds happy to add some tests so I'm returning this to him. Otherwise, I'd say it's ready for QA.

[coveralls]

coverage: 19.944% (-0.03%) from 19.969% when pulling f164a68 on 9763-lookup-optimizations into 62a9811 on develop.

[pdurbin]

I didn't run the code but it looks good. Conflicts resolved. Tests added. Docs look good.

I did notice a sql script naming conflict so I bumped the version. And I merged the latest from develop.

I'm only holding off on moving this to QA because Jenkins tests are failing (the "fire off installer" step). I started a thread in Slack about it: https://iqss.slack.com/archives/C010LA04BCG/p1697206082190619

[pdurbin]

I just kicked this off to get a second opinion from our container-based API test runner: https://github.com/gdcc/api-test-runner/actions/runs/6509472756

[pdurbin]

Jenkins is still failing but GitHub Actions passed: https://github.com/gdcc/api-test-runner/actions/runs/6509472756

Approved.

[pdurbin]

I ran Jenkins again. It's now saying [ERROR] DatasetsIT.getVersionFiles:3485 1 expectation failed.", "JSON path data[1].label doesn't match.", "Expected: test_1.txt", " Actual: test_3.txt"

@landreev should we be worried about this? From https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-9883/17/consoleFull

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:9763-lookup-optimizations

ghcr.io/gdcc/configbaker:9763-lookup-optimizations

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[pdurbin]

We're having some testing challenges within Jenkins at the moment...

• https://iqss.slack.com/archives/C010LA04BCG/p1697466092510429
• https://iqss.slack.com/archives/C010LA04BCG/p1697466281385699

... but I'm pretty sure they are unrelated to this PR. (Tests passed in GitHub Actions.) Moving to ready for QA.