Payara6: failing integration tests

[donsizemore]

Current state of integration tests on Payara 6.2023.2 running @poikilotherm's Payara6 branch:

"[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.564 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DownloadFilesIT", "[ERROR] edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8  Time elapsed: 1.033 s  <<< FAILURE!", "java.lang.AssertionError: ", "Expected status code <201> doesn't match actual status code <400>.", "", "\tat edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8(DownloadFilesIT.java:426)", "", "[INFO] ", "[INFO] Results:", "[INFO] ", 
"[ERROR] Failures: ", "[ERROR]   DatasetsIT.testArchivalStatusAPI:3021 expected:<200> but was:<403>", 
"[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>", 
"[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>", 
"[ERROR]   DataversesIT.testMalformedFacetQueryString:305", 
"[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.", "Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SearchIT.testGeospatialSearchInvalid:1267 Expected status code <400> doesn't match actual status code <500>.", "", 
"[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.", "",
"[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.", "Expected: user user60a5e34c user60a5e34c is not authorized to modify dataset with global ID doi:10.5072/FK2/9KJJRE", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>", 
"[ERROR] Errors: ", "[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF..."

[poikilotherm]

I probably need to do the rebase again. Something went sideways during the merge.

[donsizemore]

Oliver created his squeaky clean Payara 6 branch and:

"[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.51 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DownloadFilesIT", 
"[ERROR] edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8  Time elapsed: 0.891 s  <<< FAILURE!", "java.lang.AssertionError: ", "Expected status code <201> doesn't match actual status code <400>.", "", "\tat edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8(DownloadFilesIT.java:426)", "", "[INFO] ", "[INFO] Results:", "[INFO] ", 
"[ERROR] Failures: ", 
"[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>", 
"[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>", 
"[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.", "Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "",
"[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.", "Expected: user user7cd63767 user7cd63767 is not authorized to modify dataset with global ID doi:10.5072/FK2/KCABFV", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>", 
"[ERROR] Errors: ", "[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF..."

• Big sweep: migrate codebase to Jakarta EE 10 and use Payara 6 now (third iteration) #9685

[pdurbin]

Oliver updated his branch but these last two runs failed before the API tests could be executed:

• https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/45/
• https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/46/

[pdurbin]

@donsizemore and I talked about this. The API test suite is partially executing and this is flagged as a failure. This means the report never gets pushed so we have to try to figure out which API tests failed from the console log.

It's hard to read the errors from the console log.

Here's a quick one liner that got me closer to being able to read them:

curl -s https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/46/consoleText | grep "Results:" | sed 's/", /\n/g' | sed 's/^"//g'

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>
[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>
[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.

[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.
Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.
  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='

[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.

[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.
Expected: user user647feb5b user647feb5b is not authorized to modify dataset with global ID doi:10.5072/FK2/3OI2W6
  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='

[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>
[ERROR] Errors: 
[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF...
[INFO] 
[ERROR] Tests run: 197, Failures: 7, Errors: 1, Skipped: 8
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:38 min
[INFO] Finished at: 2023-07-18T22:17:31Z
[INFO] ------------------------------------------------------------------------

I'm pretty sure the same tests are failing.

[donsizemore]

@pdurbin well, the API test suite is fully executing, but we set any_errors_fatal so Ansible bails at that step, and doesn't complete the surefire reporting stuff. Without any_errors_fatal Ansible would carry on, finish its run successfully, and Jenkins would see a check. But yes, the errors will always show in the console log.

[cmbz]

Sized, labeled, and moved to Sprint Ready as per conversation with @scolapasta

[pdurbin]

We're definitely suffering from problem related to the SWORD API that we saw with the last attempted Payara 6 upgrade. Here's the commit I put in last time: d86e220

I basically flagged it as an incompatible change:

• To upload files, the SWORD API now requires "Content-Disposition: attachment; filename=example.zip" rather than "Content-Disposition: filename=example.zip".

Here are some comments I had added:

// Under Payara 5 we could send "Content-Disposition: filename=example.zip"
// Under Payara 6 now must send "Content-Disposition: attachment; filename=example.zip"
// Otherwise we get "Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='"
// Use req.getHeader("Content-Disposition") to see what the client is sending.

Here's my original writeup of the problem and the solution above: #8305 (comment)

I'll look around for any more new information but we might have to simply accept this backward incompatibility.

[pdurbin]

As I mentioned in Slack, I'm no longer sure what's going on.

• In Jenkins, we see errors that seem to implicate the platform change that breaks SWORD, as mentioned above: Payara6: failing integration tests #9457 (comment)
• On my work laptop using Docker, all SWORD tests are passing.
• On my home laptop using non-Docker, all SWORD tests are passing.
• On GitHub Actions, all SWORD tests are passing but other tests are failing (details below). We're new to running the API tests suite in GitHub Actions so a good follow up test would be to see how the "develop" branch fares in GitHub Actions.

Here are the results from the Payara 6 branch as of 26e66d8. Notice that there are a lot of 500 errors. If these 500 errors are due to Payara falling over, we might see different tests fail on a subsequent run. From https://github.com/pdurbin/dataverse-api-test-runner/actions/runs/5659573350/job/15333705090

[INFO] Results:
[INFO]
Error: Failures:
Error: AuxiliaryFilesIT.testUploadAuxFiles:69 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileExportDdi:2331 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileTermsOfUseAndAccess:2889 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFilesWORequestAccess:2960 Expected status code <409> doesn't match actual status code <400>.

Error: DataversesIT.testMalformedFacetQueryString:289
Error: DeleteUsersIT.testDeleteUserWithFileAccessRequests:517 Expected status code <200> doesn't match actual status code <400>.

Error: DownloadFilesIT.downloadAllFilesRestricted:261 Expected status code <200> doesn't match actual status code <400>.

Error: HarvestingClientsIT.testHarvestingClientRun:234 Last harvest not reported a success (took 0 seconds) expected: but was:
Error: HarvestingServerIT.testMultiRecordOaiSet:595->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: HarvestingServerIT.testSingleRecordOaiSet:443->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: LinkIT.testDeepLinks:169 Expected status code <200> doesn't match actual status code <500>.

Error: MakeDataCountApiIT.testMakeDataCountGetMetric:61 Expected status code <200> doesn't match actual status code <400>.

Error: SearchIT.testGeospatialSearch:1233 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testGeospatialSearchInvalid:1274 Expected status code <400> doesn't match actual status code <500>.

Error: SearchIT.testIdentifier:753 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testNestedSubtree:801 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testSubtreePermissions:987 Expected status code <200> doesn't match actual status code <500>.

Error: Errors:
Error: DataversesIT.testImportDDI:536 » IllegalArgument Cannot get property 'total_co...
Error: DataversesIT.testMoveDataverse:380 » IllegalArgument Cannot get property 'tota...
Error: FilesIT.testAccessFacet:1111 » IllegalArgument Cannot get property 'total_coun...
Error: SearchIT.testAdditionalDatasetContent6300:243 » IllegalArgument Cannot get pro...
Error: SearchIT.testCuratorCardDataversePopulation:917 » IllegalArgument Cannot get p...
Error: SearchIT.testDatasetThumbnail:346 » IllegalArgument Cannot get property 'items...
Error: SearchIT.testSearchCitation:183 » IllegalArgument Cannot get property 'items' ...
Error: SearchIT.testSearchDynamicMetadataFields:292 » IllegalArgument Cannot get prop...
Error: SearchIT.testSearchPermisions:77 » IllegalArgument Cannot get property 'total_...
[INFO]
Error: Tests run: 209, Failures: 17, Errors: 9, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:55 min
[INFO] Finished at: 2023-07-25T17:18:34Z
[INFO] ------------------------------------------------------------------------

Update, here another GitHub Actions run, from https://github.com/pdurbin/dataverse-api-test-runner/actions/runs/5660040551/job/15334873565

[INFO] Results:
[INFO]
Error: Failures:
Error: AuxiliaryFilesIT.testUploadAuxFiles:69 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileExportDdi:2331 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileTermsOfUseAndAccess:2889 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFilesWORequestAccess:2960 Expected status code <409> doesn't match actual status code <400>.

Error: DataversesIT.testMalformedFacetQueryString:289
Error: DeleteUsersIT.testDeleteUserWithFileAccessRequests:517 Expected status code <200> doesn't match actual status code <400>.

Error: DownloadFilesIT.downloadAllFilesRestricted:261 Expected status code <200> doesn't match actual status code <400>.

Error: HarvestingClientsIT.testHarvestingClientRun:234 Last harvest not reported a success (took 0 seconds) expected: but was:
Error: HarvestingServerIT.testMultiRecordOaiSet:595->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: HarvestingServerIT.testSingleRecordOaiSet:443->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: LinkIT.testDeepLinks:169 Expected status code <200> doesn't match actual status code <500>.

Error: MakeDataCountApiIT.testMakeDataCountGetMetric:61 Expected status code <200> doesn't match actual status code <400>.

Error: SearchIT.testGeospatialSearch:1233 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testGeospatialSearchInvalid:1274 Expected status code <400> doesn't match actual status code <500>.

Error: SearchIT.testIdentifier:753 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testNestedSubtree:801 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testSubtreePermissions:987 Expected status code <200> doesn't match actual status code <500>.

Error: Errors:
Error: DataversesIT.testImportDDI:536 » IllegalArgument Cannot get property 'total_co...
Error: DataversesIT.testMoveDataverse:380 » IllegalArgument Cannot get property 'tota...
Error: FilesIT.testAccessFacet:1111 » IllegalArgument Cannot get property 'total_coun...
Error: SearchIT.testAdditionalDatasetContent6300:243 » IllegalArgument Cannot get pro...
Error: SearchIT.testCuratorCardDataversePopulation:917 » IllegalArgument Cannot get p...
Error: SearchIT.testDatasetThumbnail:346 » IllegalArgument Cannot get property 'items...
Error: SearchIT.testSearchCitation:183 » IllegalArgument Cannot get property 'items' ...
Error: SearchIT.testSearchDynamicMetadataFields:292 » IllegalArgument Cannot get prop...
Error: SearchIT.testSearchPermisions:77 » IllegalArgument Cannot get property 'total_...
[INFO]
Error: Tests run: 209, Failures: 17, Errors: 9, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:49 min
[INFO] Finished at: 2023-07-25T18:04:03Z
[INFO] ------------------------------------------------------------------------