add S3 tests, LocalStack, MinIO

[pdurbin]

What this PR does / why we need it:

Developers can now test S3 locally by using the Dockerized development environment, which now includes both LocalStack and MinIO. See S3AccessIT which executes API (end to end) tests.

In addition, a new integration test test class
(not an API test, the new kind launched with mvn verify) has been added at S3AccessIOLocalstackIT. It uses Testcontainers to spin up Localstack for S3 testing and does not require Dataverse to be running.

TODO: Exercise direct upload, possibly borrowing code from DVUploader.

Which issue(s) this PR closes:

• Closes Add S3 tests to the regular integration test suite #6783

Special notes for your reviewer:

Note that the format of docker-compose-dev.yml had to change to allow for JVM options to be added.

Suggestions on how to test this:

mvn -Pct clean package docker:run

mvn test -Dtest=S3AccessIT

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No.

Is there a release notes update needed for this change?:

Yes, included.

Additional documentation:

Tweaked various docs:

• Listing and setting stores via API.
• Using the direct upload API.

[coveralls]

coverage: 20.174% (+0.1%) from 20.037%
when pulling cff73b8 on 6783-s3-tests
into d514545 on develop.

[donsizemore]

@jp-tosca I got things (mostly) working this morning; discovered what we believe is a bug in Dataverse in the process. Just wanted to document here that, while MinIO doesn't like repeated attempts to create a bucket via AWS CLI, re-running S3AccessIt tests manually doesn't return an error. Rocky Linux 8, LocalStack and MinIO are Dockerized.

[donsizemore]

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:6783-s3-tests

ghcr.io/gdcc/configbaker:6783-s3-tests

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[pdurbin]

After connecting to the VPN the tests run fine.

I just tried mvn test -Dtest=S3AccessIT from home and I'm not on the VPN. The tests pass fine:

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[pdurbin]

I approved the PR on the Ansible side:

• support Dockerized s3 tests gdcc/dataverse-ansible#337

Thank you, @donsizemore for all the hard work! 🚀 ❤️ 🎉

Once it's merged, we'll kick off another build to see if Jenkins starts passing.

[donsizemore]

Once it's merged, we'll kick off another build to see if Jenkins starts passing.

@pdurbin the Dataverse-Ansible branch is merged, so if you're comfortable with your own testing, Jenkins is ready.

[jp-tosca]

Thanks a lot @donsizemore !!

[pdurbin]

@donsizemore thanks! I just kicked off a build: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-10044/17/