9434 app container for dev by poikilotherm · Pull Request #9439 · IQSS/dataverse

poikilotherm · 2023-03-13T16:06:17Z

What this PR does / why we need it:

This pull request will add capabilities to create a Dataverse application image and run it along all necessary dependencies.

Which issue(s) this PR closes:

Closes Build locally a Dataverse app image for dev #9434

Special notes for your reviewer:
There are some TODOs:

Get Dataverse communicating with maildev properly
~~(Optional) Maybe add some more resource limitations. Maybe tweak Dataverse container to fit into less than 2 GiB of RAM~~
Add docs (can be copied from 9415 containers #9424 and extend from there). DONE.

Suggestions on how to test this:
Clone/switch. Ensure to have Docker running. Execute mvn -Pct clean package docker:run, then bootstrap when startup is done with ./scripts/dev/docker-final-setup.sh and go to localhost:8080.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope.

Is there a release notes update needed for this change?:
Yes, included.

Additional documentation:
None yet.

…operties

…ir env var

…hecked

pdurbin · 2023-03-14T01:35:54Z

@poikilotherm this is working great!

Thanks for the heads up about email. I tried using the "Support" link in the header but when I clicked "Send Message" I got this error:

dev_dataverse> Caused by: java.lang.NullPointerException
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.setContactDelegation(MailServiceBean.java:233)

Honestly, I can live without email working but it would be nice to see if we can fix this.

I can also live without the TODO above about limiting resources (for now). Future PR, I'd say. I marked it as optional.

I added docs and a release note. I tried to indicate that this is dev only.

Hopefully the last thing is the SMTP thing. Or, again, I'm fine with fixing later.

There is a TON of value in this PR. Thanks!!

pdurbin · 2023-03-14T11:41:27Z

dev_dataverse> Caused by: java.lang.NullPointerException
dev_dataverse> at edu.harvard.iq.dataverse.MailServiceBean.setContactDelegation(MailServiceBean.java:233)

Here's a stacktrace of the MailDev/SMTP problem: stacktrace.txt

Apparently, fromAddress here is null:

dataverse/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java

Line 233 in a7b0457

String personal = fromAddress.getPersonal() != null

In the domain.xml, the from address is dataverse@localhost:

payara@dataverse:~$ grep mail-resource /opt/payara/appserver/glassfish/domains/domain1/config/domain.xml
<mail-resource auth="false" host="postfix" from="dataverse@localhost" user="dataversenotify" jndi-name="mail/notifyMailSession"></mail-resource>

The from address is not defined as a database setting:

payara@dataverse:~$ curl -s http://localhost:8080/api/admin/settings | jq .
{
  "status": "OK",
  "data": {
    ":Authority": "10.5072",
    ":Shoulder": "FK2/",
    "BuiltinUsers.KEY": "burrito",
    ":DoiProvider": "FAKE",
    ":BlockedApiPolicy": "localhost-only",
    ":Protocol": "doi",
    ":AllowSignUp": "yes",
    ":SignUpUrl": "/dataverseuser.xhtml?editMode=CREATE",
    ":UploadMethods": "native/http"
  }
}

Setting it might fix it.

If I set :SystemEmail like this...

$ curl -X PUT -d 'dataverse@localhost' http://localhost:8080/api/admin/settings/:SystemEmail
{"status":"OK","data":{":SystemEmail":"dataverse@localhost"}}

I get a new error:

dev_dataverse> [#|2023-03-14T11:51:01.758+0000|INFO|Payara 5.2022.4||_ThreadID=87;_ThreadName=http-thread-pool::http-listener-1(3);_TimeMillis=1678794661758;_LevelValue=800;|
dev_dataverse>   com.sun.mail.util.MailConnectException: Couldn't connect to host, port: postfix, 25; timeout -1;
dev_dataverse>   nested exception is:
dev_dataverse> 	java.net.ConnectException: Connection refused (Connection refused)
dev_dataverse> 	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2209)
dev_dataverse> 	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:342)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:222)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:171)
dev_dataverse> 	at javax.mail.Transport.send0(Transport.java:230)
dev_dataverse> 	at javax.mail.Transport.send(Transport.java:100)
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.sendMail(MailServiceBean.java:217)
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.sendMail(MailServiceBean.java:181)

From my Mac, the SMTP server is running on port 25...

pdurbin@air dataverse % nc localhost 25
220 postfix ESMTP

... but the dataverse-1 (app container) can't reach it:

payara@dataverse:~$ nc postfix 25
payara@dataverse:~$

Why?

docker-compose-dev.yml

GPortas · 2023-03-14T13:39:45Z

@poikilotherm

I have done the following steps so far:

Built the image through mvn -Pct clean package
Executed the containers through docker-compose -f docker-compose-dev.yml up
Executed ./scripts/dev/docker-final-setup.sh when startup is done

Received this error within the docker-compose logs while applying the third step:

dev_dataverse_1         | [#|2023-03-14T13:25:45.002+0000|WARNING|Payara 5.2022.4|javax.enterprise.ejb.container|_ThreadID=81;_ThreadName=http-thread-pool::http-listener-1(1);_TimeMillis=1678800345002;_LevelValue=900;|
dev_dataverse_1         |   javax.ejb.EJBException: getSingleResult() did not retrieve any entities.
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.processSystemException(EJBContainerTransactionManager.java:723)
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:652)
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:482)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4601)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2134)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2104)

. . .

Dataverse runs on 8080, but when trying to Log In with dataverseAdmin / admin credentials, the authentication fails. Am I missing something?

poikilotherm · 2023-03-14T14:10:45Z

@GPortas please note that the login is "dataverseAdmin:admin1". That "1" is easy to forget.

Also, the "getSingleResult()" thing is very annoying but harmless. For some reason no one ever fixed this also it would be a low hanging fruit. Same goes for all the SQL exceptions which are triggered by us generating the DDL everytime we start the application.

All containers have static names now, which might be a problem down the road. But at least consistent now.

Now Maildev will listen on port 25 as necessary to let Dataverse reach it. Also adapted the storage path to be easier addressable with a volume. Switched to using a tmpfs as other another initializer would be necessary to switch the volume permissions (like with Solr). Added configuring the SystemEmail to match the default value from init_2_config_payara.sh. Otherwise no mail gets sent.

pdurbin · 2023-03-14T18:22:01Z

javax.ejb.EJBException: getSingleResult() did not retrieve any entities.

This happens with every installation of Dataverse, so it has nothing to do with Docker. It's caused by this code:

/**
 * @todo Move this to
 * AuthenticationServiceBean.createAuthenticatedUser
 */
boolean rootDataversePresent = false;
try {
    Dataverse rootDataverse = dataverseSvc.findRootDataverse();
    if (rootDataverse != null) {
        rootDataversePresent = true;
    }
} catch (Exception e) {
    logger.info("The root dataverse is not present. Don't send a notification to dataverseAdmin.");
}
if (rootDataversePresent && sendEmailNotification) {
    userNotificationSvc.sendNotification(au,
            new Timestamp(new Date().getTime()),
            UserNotification.Type.CREATEACC, null);
}

Which is triggered by creating the dataverseAdmin user:

echo "Setting up the admin user (and as superuser)"
adminResp=$(curl -s -H "Content-type:application/json" -X POST -d @data/user-admin.json "$SERVER/builtin-users?password=$DV_SU_PASSWORD&key=burrito")
echo $adminResp
curl -X POST "$SERVER/admin/superuser/dataverseAdmin"

Then the dataverseAdminUser creates the root collection and the error stops happening for future users. Let's please treat this as out of scope. Again, it's an old bug.

johannes-darms · 2023-03-16T08:16:06Z

doc/sphinx-guides/source/container/app-image.rst

+        be sent.
+
+
+Note that the script ``init_2_configure.sh`` will apply a few very important defaults to enable quick usage


I'm not sure is this section, with implementation details, contains necessary information for someone using the image.

Well this becomes very important if you configure the app to use some S3 storage... By switching the default storage to something other than "local", the local file storage will vanish.

Right now the target user is a developer, someone hacking on the Dataverse software itself. I think these developers have enough information, right? Can we pick up this thread in a future pull requests when there is a different, non-developer audience?

johannes-darms · 2023-03-16T08:18:04Z

doc/sphinx-guides/source/container/dev-usage.rst

+Development Usage
+=================
+
+Please note! This Docker setup is not for production!


If it cannot be used in production, please provide some insight why that is not a good idea.

I disagree. This section has a heading "Development Usage" and is targeting someone hacking on Dataverse and who wants to testrun changes in containers rather than a local installation of all components.

Please note! This Docker setup is not for production! 😄

We could add, "You're in the dev-usage doc!" 😄

We are deliberately walking before we run. 🚶 🏃

johannes-darms · 2023-03-16T08:20:23Z

doc/sphinx-guides/source/container/dev-usage.rst

+Intro
+-----
+
+Assuming you have `Docker <https://docs.docker.com/engine/install/>`_, `Docker Desktop <https://www.docker.com/products/docker-desktop/>`_,


Could we just link to the OCI container spec (https://opencontainers.org/) without endorsing one implementation?

No we can't. We can add Rancher Desktop if you want. Other than that, the Docker Maven Plugin requires using some sort of Moby daemon (Docker is a supported, paid distribution of Moby). You will not be able to build an image with podman or JIB etc with the current configuration.

It also is important to keep in mind that a future vision is to use these images with Testcontainers for more extensive tests than available now. This will also require some kind of Docker/Moby.

It would be nice to have this information in the documentation.

@johannes-darms sounds fine but this PR is in "Ready for QA" and it would be a shame to lose momentum. If you can make a PR into this PR, I'm happy to look and hopefully merge.

johannes-darms · 2023-03-16T08:22:34Z

doc/sphinx-guides/source/developers/containers.rst

+-------------------

-See also the :doc:`/container/index`.
+The :doc:`testing` section mentions using docker-aio for integration tests. We do not plan to keep this project alive.


Depreciation plan or end of life notice would be nice.

I'm not sure I get you here. Do you want such a notice in here, right now? Or is this something you'd like to see in the future?

For the sake of having the feature merged, I rather say in the future. However, saying that there are no plans on keeping docker-aio alive without given an EOL date isn't that nice to users of the package.

Oh! Is anyone (except me, occasionally) using docker-aio?!?

I have used docker-aio occasionally like you @pdurbin (Following the testing documentation, as it recommends to use it). I agree to point out the deprecation warning and describe it in any doc page that mentions docker-aio, to make the documentation more consistent.

johannes-darms · 2023-03-16T08:27:21Z

doc/sphinx-guides/source/container/dev-usage.rst

+
+``mvn -Pct clean package``
+
+Now, start all the containers with a single command:


Mention that this only starts the dataverse container, and up and running dependencies (postgres,SMTP and solr) are needed.

Huh? Did you test this? Because this should start all the dependencies, too, as configured in the compose file. Let me know if this doesn't work for you.

Yes, works on my machine! Containers go! 🤖 🤖 🤖 🤖

it also works for me!

doc/sphinx-guides/source/developers/containers.rst

johannes-darms · 2023-03-16T08:31:27Z

doc/sphinx-guides/source/developers/dev-environment.rst

 To avoid having to install service dependencies like PostgreSQL or Solr directly on your localhost, there is the alternative of using the ``docker-compose-dev.yml`` file available in the repository root. For this option you need to have Docker and Docker Compose installed on your machine.

-The ``docker-compose-dev.yml`` file runs the necessary service dependencies to support a development Dataverse installation running on localhost. In addition to PostgreSQL and Solr, it also runs a SMTP server.
+The ``docker-compose-dev.yml`` can be configured to only run the service dependencies necessary to support a Dataverse installation running directly on localhost. In addition to PostgreSQL and Solr, it also runs a SMTP server.


How can I configure that?

This is described below. Could you make a comment with a suggestion (Ctrl+g) on how to phrase this better? Thx!

docker-compose-dev.yml

johannes-darms · 2023-03-16T09:18:44Z

The approach is also missing R and Rserve. They are listed as a prerequisite within the documentation but it looks like they are not really needed. If R/RServe is optional the documentation must be updated, otherwise another container wrapping RServe is needed.

https://guides.dataverse.org/en/latest/installation/prerequisites.html?highlight=rserve

poikilotherm · 2023-03-16T09:33:25Z

Thanks @johannes-darms for the extensive review of the PR! I added comments to your comments.

The approach is also missing R and Rserve. They are listed as a prerequisite within the documentation but it looks like they are not really needed. If R/RServe is optional the documentation must be updated, otherwise another container wrapping RServe is needed.

https://guides.dataverse.org/en/latest/installation/prerequisites.html?highlight=rserve

Two things here: 1) Rserve is not a necessary component for developing Dataverse, which is what this is about. 2) Technically it even is not a requirement for an installation if you don't do any ingest. But yes, that should be stated in the docs. Would you be able to create an issue and/or a pull request?

pdurbin · 2023-03-16T19:22:59Z

not a necessary component for developing Dataverse, which is what this is about

Good point. @poikilotherm how do you feel about a new title for this PR?

"app container" -> "app container for dev"

@johannes-darms thanks for all the feedback! I also left comments on your comments. Much appreciated!!

GPortas · 2023-03-17T12:43:11Z

@poikilotherm

Although the asadmin commands are enough to manage the Payara server, sometimes it is very useful to access the Payara administration console for better visualization.

By default, Payara exposes the administration console on port 4848. It would be interesting to map this port to outside the container within the docker-compose file.

I've tested this with this branch, by adding - "4848:4848" to the dev_dataverse mapped ports.

The problem is that the admin console now expects user and password credentials to login (As far as I know, these credentials are not documented). With the normal installation (without containers), we don't need credentials for accessing the Payara admin console. Is that login requirement a configuration that comes from the base image? It would be great to make admin credentials configurable through container environment variables.

poikilotherm · 2023-03-17T12:48:44Z

@GPortas yes, this comes from the base image. We could add a section to the base image documentation, similar to upstream Payara as this works exactly the same in our custom base image.

In fact, the credentials are available via the (currently undocumented) env vars ADMIN_USER and ADMIN_PASSWORD, but changing these env vars doesn't update the running domain.

@GPortas probably this is beyond scope for this PR. Would you add a note to https://docs.google.com/document/d/15-sqdKzpCgQBtaPaAGYMaqcxsQXmt-2CwAVNPmmAPRY so we don't forget?

poikilotherm · 2023-03-17T13:47:13Z

@pdurbin @johannes-darms I added a few pieces of information about a future sunset of docker-aio in 7bcadab. Let me know if that helps. Feel free to hack on it.

pdurbin · 2023-03-17T18:10:24Z

a future sunset of docker-aio

Moving docker-dcm to this repo sbgrid/data-capture-module#39

kcondon · 2023-03-20T14:17:45Z

Issues so far:

Running into build failure due to test failures. Am I doing something wrong?
app_contain_bld_fail.txt
[Kevin] I think it was because I didn't do switch -c branch, gets further now, see issue 2
Mvn script doesn't complete, pauses, shows errors. Ran set up script and localhost:8080 shows only Payara.
mvn_err_pause.txt
final_setup_run.txt

poikilotherm · 2023-03-20T15:24:45Z

Hi @kcondon, thx for trying this. The log says that the database connection could not be established. It looks like you already had the local volumes from before (most likely #9414 / #9417), so there might be a different database already in place. Could you delete the whole ./docker-dev-volumes folder and try again? Thx!

Without setting the "env var" for docker compose within the Maven POM, the value will not be set, leading to a broken DB connection. The DMP does not read the values from .env.

poikilotherm · 2023-03-20T19:51:37Z

Woo-hoo thanks for merging @kcondon ! Much appreciated, as always!!

poikilotherm added 8 commits March 13, 2023 15:11

fix(ct): make directory config source override microprofile-config.pr…

76f1a74

…operties

chore(deps): update docker maven plugin to 0.42.0

519b7d0

feat(ct): make temp file storage location in containers use storage d…

65f2478

…ir env var

feat(ct): add configure script hack for Payara to get going

7374bb7

feat(ct): add application Dockerfile and assembly config

21e8d72

feat(ct): add cap to build images for the application from Maven

5f49e68

feat(ct): extend compose file to include and start application

dde54df

feat(ct): add simple bootstrap script inspired by @pdurbin but shellc…

b052423

…hecked

poikilotherm requested review from GPortas and pdurbin March 13, 2023 16:06

poikilotherm self-assigned this Mar 13, 2023

This was referenced Mar 13, 2023

9415 containers #9424

Closed

Analyze Dataverse application containerization for development #9415

Closed

Build locally a Dataverse app image for dev #9434

Closed

pdurbin unassigned poikilotherm Mar 13, 2023

add docs, rename script #9434

a7b0457

pdurbin self-assigned this Mar 14, 2023

GPortas self-assigned this Mar 14, 2023

GPortas reviewed Mar 14, 2023

View reviewed changes

docker-compose-dev.yml Outdated Show resolved Hide resolved

poikilotherm added 2 commits March 14, 2023 15:25

fix(ct): add missing DV container name in docker compose file

47d6ed4

All containers have static names now, which might be a problem down the road. But at least consistent now.

johannes-darms reviewed Mar 16, 2023

View reviewed changes

docker-compose-dev.yml Show resolved Hide resolved

poikilotherm changed the title ~~9434 app container~~ 9434 app container for dev Mar 16, 2023

doc(ct): add note about future sunset of docker-aio

7bcadab

pdurbin mentioned this pull request Mar 17, 2023

Moving docker-dcm to this repo sbgrid/data-capture-module#39

Open

feat(ct): make app image name overrideable from Maven build

bf51294

kcondon assigned kcondon and unassigned kcondon Mar 20, 2023

fix(ct): add missing db user var to POM

b99073c

Without setting the "env var" for docker compose within the Maven POM, the value will not be set, leading to a broken DB connection. The DMP does not read the values from .env.

kcondon merged commit a9a8311 into develop Mar 20, 2023

kcondon deleted the 9434-app-container branch March 20, 2023 19:17

pdurbin mentioned this pull request Mar 24, 2023

Support for deleting files using native API #9383

Merged

GPortas mentioned this pull request Mar 27, 2023

Make Payara Admin Console accessible in the containerized setup #9476

Closed

GPortas mentioned this pull request Apr 12, 2023

Solr custom Docker image for dev #9516

Closed

pdurbin added this to the 5.14 milestone May 10, 2023

helkv mentioned this pull request Aug 16, 2023

Edmond-Dataverse in Docker MPDL/dataverse#71

Open

cmbz added FY26 Sprint 14 FY26 Sprint 14 (2025-12-31 - 2026-01-14) and removed FY26 Sprint 14 FY26 Sprint 14 (2025-12-31 - 2026-01-14) labels Jan 14, 2026

		be sent.


		Note that the script ``init_2_configure.sh`` will apply a few very important defaults to enable quick usage


		``mvn -Pct clean package``

		Now, start all the containers with a single command:

Comments

Conversation

poikilotherm commented Mar 13, 2023 • edited by pdurbin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pdurbin commented Mar 14, 2023

Uh oh!

pdurbin commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

GPortas commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

poikilotherm commented Mar 14, 2023

Uh oh!

pdurbin commented Mar 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

poikilotherm Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johannes-darms Mar 16, 2023 • edited by pdurbin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johannes-darms Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdurbin Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GPortas Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johannes-darms commented Mar 16, 2023

Uh oh!

poikilotherm commented Mar 16, 2023

Uh oh!

pdurbin commented Mar 16, 2023

Uh oh!

GPortas commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

poikilotherm commented Mar 13, 2023 •

edited by pdurbin

Loading

pdurbin commented Mar 14, 2023 •

edited

Loading

GPortas commented Mar 14, 2023 •

edited

Loading

poikilotherm Mar 16, 2023 •

edited

Loading

johannes-darms Mar 16, 2023 •

edited by pdurbin

Loading

johannes-darms Mar 16, 2023 •

edited

Loading

pdurbin Mar 16, 2023 •

edited

Loading

GPortas Mar 16, 2023 •

edited

Loading

GPortas commented Mar 17, 2023 •

edited

Loading

poikilotherm commented Mar 17, 2023 •

edited

Loading

kcondon commented Mar 20, 2023 •

edited

Loading

poikilotherm commented Mar 20, 2023 •

edited

Loading