Proposal: make persistent EJB timers non-persistent or not rely on database

[poikilotherm]

This is related to #5292 and a story of that epic.
As DataverseTimerServiceBean.java has been mostly written and maintained by @landreev, this most certainly is for him...

Context

Currently, in glassfish-setup.sh you need to setup a database connection for storage of persistent EJB timers.

Reusing the database for Dataverse for EJB timers is alright, but it is not easy to depict when heading for containers. It is much easier for container approaches to use application scoped JDBC connections, but those seem not to be reusable for EJB timers.

There seemed to be some issues in the past, too. #3672, #3789 and more.

Proposal

Just remove the persistent nature of timers.

• They are created on each startup on the timing server (see docs).
• In a single node setup, when the server is down, IMHO there is no need to have the timers persistent in a database. No one will execute them. Executing them after the server comes back seems with HarvestingClient and ExportClient as current use cases not very important.
• In a multi node setup, when the timing server is down, you need to change it to another server, so they get executed again. This will result in broken persistent timers anyway, as they had been created with the name of the dead timing server.

When persistance is needed in a multi node setup in the future, there is Hazelcast in Payara which supports storing timers as long as at least one cluster member is online...

Comments are very much appreciated. :-)

[poikilotherm]

Following @pdurbin suggestion about DDD (doc driven development), I request a review of my plans in #5371 by @landreev and potentially others.

Cheers 😄 🍺

[landreev]

@poikilotherm
As a quick note, yes, this makes perfect sense.

At present, we are not benefiting from the fact that the timers are stored in the database, at all. It's kind of the other way around: you can say that we go to the extra trouble of making sure that our timers are NOT persisted. In a multi-node cluster, we specifically try to run all the timers on just one of the nodes (for various reasons); AND we clear and re-create all the timers on startup.

So yes, it will be cleaner, not to bother saving them in the database in the first place.

(we actually have tried something similar, in our own production cluster at Harvard a while ago; I can't remember now why we chose not to make it the default solution in the distribution...)

I may have questions about some specifics of the PR (#5371) though; I'll be in touch.

[poikilotherm]

Hi @landreev, @pdurbin told me you guys talked about this and you seemed to like it. Should I start actual code work on this?

[poikilotherm]

Hey @landreev,

could you give my comments in 81fbffe a look and discuss this with others? I am uncertain if this refactoring is a chunk to big to fit inside this PR. (Happy to do the necessary code work).

And while you are in there, you could take a look at a7cc7b8 which puts the first (but not all) @Schedule annotations in place.

Thx!