Problems with the EJB Timer (in production environment, specifically) #3672
Description
This must be/probably is related to the issue where the EJB Timer's lock on the database prevents Glassfish to start and/or the application to deploy (for example, #3669)
The issue with the timer, currently observed on our prod. dedicated timer service (dvn-sum1-app-2/picard), is that it just stops working completely. Even the top-level, master timer stops firing - and then none of the scheduled harvests and exports are happening.
We need to finally figure out what is going on with that timer. Part of the difficulty with diagnosing it is that the timer is a standalone EJB app, supplied with Glassfish. But it should still be possible to obtain its source and see what's going on there - if everything else fails.
An interesting observation is that nobody has ever seen these timer issues in their dev. environments. It only happens on "real" servers... but what does that mean exactly? - could be as trivial as Mac OS vs. Linux. Or is it something about running the database on loacalhost vs. over a non-local network, with more ports firewalled?