Handling OMERO and Bio-Formats Java dependencies

[chris-allan]

Our team has been plowing through various integration work with CellProfiler, Bio-Formats and OMERO. This involves utilizing the existing CellProfiler-OMERO integration in both a classical user focused, graphical user interface context and also a headless, run from OMERO.scripts context both for ourselves and on behalf of our users.

While running pipelines in anger and on large datasets (plates in the neighborhood of 384 well, 10 field, 2048x2048, 5 channels as an example), headless or not, we have come across some bugs in OmeroReader that need to be fixed. This class comes from OMERO's Blitz.jar and is currently bundled into the prokaryote mega JAR. As CellProfiler communicates with OMERO through a CellProfiler (Python) -- javabridge --> Bio-Formats (Java) --> OMERO (Java) conduit, we need ways to affect and confidently state to CellProfiler which JARs we want used when CellProfiler extracts pixel data from OMERO. We've played a lot with this over the past few months.

This is where we start running into issues affecting the CellProfiler CLASSPATH. Recently we saw #1733 go in which understandably attempts to manage as many of the Java dependencies that CellProfiler uses under one roof via PyPi. Completely sensible and much appreciated! Unfortunately that has the downstream effect of putting these dependencies into a singular walled garden that we cannot affect save blowing it away entirely and explicitly defining, on our own CLASSPATH, all the JARs we want to use. This is compounded by the fact that JARs on the CLASSPATH in the current user's environment are appended to the CLASSPATH that CellProfiler ends up using. Thus resulting in an inability to, via the return first match semantics of Java CLASSPATH resolution, replace one or more classes, or even entire JARs, whose classes are present in the mega JAR.

Furthermore, the versions and transitive dependencies that are pulled in are fixed. Using OMERO as an example, these are pinned to OMERO 5.1.3-ice35-b52 at the time of this writing. OMERO 5.1.4 was released in September 2015, 5.2.0 in November and 5.2.1 in December. That the major versions, at a minimum, of these client JARs match with those of the server is essential to the function of CellProfiler-OMERO integration. What we are not suggesting is that it is the job of the CellProfiler project to come up with at solution to manage these version differences. Again, we just need a way to robustly, properly, and with the CellProfiler project's knowledge state which versions we want used.

Our team are quite prepared to scratch our own itch and contribute to resolving these issues but the number of moving parts here is quite substantial. Especially in light of CellProfiler's direct dependency on Bio-Formats. As far as the CellProfiler codebase is concerned we're feeling a bit out of our depth. Before we embark on a pull request, a few questions:

• Are we completely misrepresenting the situation? Is there something essential that we have overlooked?
• Should we bring the user's environmental CLASSPATH to the front of the line via changes to cpjvm.py?
• What are your thoughts about OMERO and Bio-Formats version management in the context of prokaryote.jar? Should "advanced" users just delete this and manage the versions of these dependencies ourselves via our own CLASSPATH environment variables?

Thanks!

/cc @emilroz, @joshmoore, @jburel, @sbesson, @bramalingam

[LeeKamentsky]

We could use some help in this regard. Packaging all of the jars into one solved some deployment problems for us and having our java dependencies in a separate repository has helped simplify our build process.

Nevertheless, we've had severe difficulties figuring out the interaction between the classpath variable, the class loader and our dependencies and we may be a bit out of our league. I would appreciate some help from you in this regard and would welcome a pull request. Indeed, my expectation was that I would be able to place the jars on the user's classpath ahead of prokaryote.jar and have them take precedence and I was quite crushed when my experiments seemed to show otherwise.

We've gone back and forth on Maven. Internally, it's a great tool and I wish we had something like it in Python, but we also feel that, in the Python ecosystem, it poses a burden on developers to potentially have to understand it. My expectation was that the OMERO dependencies could be handled by jars on the command line, although it does make me nervous attempting to solve the dependency problems outside of Maven.

Going forward, I could see a couple of solutions:

• Expecting users delete prokaryote.jar and substitute their own jars, possibly with the help of a pom.xml file that they could modify and run.
• Versions of prokaryote.jar matched to various ImageJ and BioFormats releases
• A hybrid solution where BioFormats dependencies were delivered separately from our prokaryote.jar
• Making cpjvm capable of preferentially loading from the user's classpath
• Making it possible for the user to substitute prokaryote.jar with the old-style mechanism (all Jars appearing in a certain folder with a text file that gives the order of the jars in the classpath
• Returning to the old-style mechanism completely: download a copy of Maven, run Maven against our POM to create or download the jars and to create the text file giving the classpath.

/cc @chris-allan @emilroz @joshmoore @jburel @sbesson @bramalingam @0x00b1

[chris-allan]

Thanks for the feedback, @LeeKamentsky.

After some discussion, @emilroz has come up with a short term solution (#1773) that will at least allow for the environmental override, via CLASSPATH, of certain JARs. Your "making cpjvm capable of preferentially loading from the user's classpath" solution in a nutshell. We are, in our case, namely interested in those applying to OMERO but this should work for any JAR(s).

I should add that using the above solution for major version switches, ie. OMERO 5.1.x to 5.2.x, would be incredibly dangerous. It's just not possible to really resolve all the potential transient dependency issues that you may have.

Discussing some of these dependency management options with @melissalinkert, we think that a documented pom.xml based workflow with one or more sets of dependencies either included and commented out or explicitly separated by branch would probably work best.

The only downside of this is that it's not very pythonic. It would likely mean that pip install prokaryote would possibly then be followed by a download or clone of the prokaryote repository, editing of the pom.xml, running of Maven and then a replacement of prokaryote.jar.

You could imagine a series of versions and/or suffixes such as 1.0.8-omero5-bf51 or prokaryote-omero5-bf51 being released to PyPi. However, I can quickly see this spiraling out of control as far as testing is concerned and it requires you to be intimately knowledgeable about which versions of the major dependencies work together and potential conflicts. Not something I would wish on anyone.

As a sidenote, Bio-Formats does have an impending version decoupling coming and it will no longer be sufficient to assume that Bio-Formats 5.2.x works with all OMERO 5.2.x releases.

On that topic, in light of the dependencies, and the increasing burden that is placed on all of us by them we'd like to propose that our team pursue transitioning the CellProfiler-OMERO integration to pure Python. OMERO has excellent Python bindings and it seems a real shame not to be using them while at the same time saddling ourselves with managing the trifecta of Bio-Formats, OmeroReader and OMERO compatibility. This would, in our opinion, have a few obvious near term advantages:

• Better performance
• Lower memory usage
• Cleaner codebase
• Removal of OMERO Java dependencies from those that CellProfiler actively has to track

Let us know your thoughts.

[chris-allan]

During the past few months since we originally raised this issue @glencoesoftware and various members of the @openmicroscopy community have been running CellProfiler in production making heavy use of the CellProfiler-OMERO integration. This has been predominantly, but not exclusively, in headless mode at a wide variety of scales in a variety of deployment environments. Usually a cluster has been involved. One such deployment and execution environment was outlined in some detail during a @glencoesoftware webinar back in May:

• http://glencoesoftware.com/webinars.html#omero-plus-for-hcs

Most of the overarching deployment environments we have been exposed to are mixed. Where the goal in almost all cases is to use the CellProfiler-OMERO integration both during pipeline development and larger scale, headless, parallel execution on a cluster. Pipeline development is often being performed by biologists who inevitably end up utilizing your provided executables to get CellProfiler installed on standalone workstations. The headless, parallel execution environments are installed from source and are heavily managed.

Given this we feel it is an opportune time to revisit this issue in more detail backed by this real world experience and come up with better short and medium term solutions to using CellProfiler with OMERO.

Having CellProfiler-OMERO integration “baked in” has been a bit of a double edged sword for us during this time. Mostly for the aforementioned issues:

• Mismatches in actual required OMERO client library version vs. that provided by prokaryote / prokaryote.jar; mostly limited to the standalone workstation environment, in the headless, parallel environment we are utilising the fix from Import CLASSPATH deps first (See #1772) #1773 and managing the JARs explicitly
• Insufficient OMERO client library version provided by prokaryote / prokaryote.jar; currently set at 5.2.2-ice35-b17 and last updated on April 23rd, there have been three major releases of OMERO since
• Insufficient Bio-Formats library version provided by prokaryote / prokaryote.jar; currently set at 5.1.9 last updated on April 23rd, there has been one minor release of Bio-Formats since

We’ve also run up against a couple of new related issues:

• Ice 3.6 support in OMERO from version 5.2.3+ leading to more JAR requirement variability
• Unnecessary dependencies of OmeroReader introduced due to the deep requirements of blitz.jar

The OME team is aware of the latter and is doing its utmost to address it in the upcoming 5.3.0 release:

• https://trello.com/c/b8jrapBD/174-omero-reader

During assay development with a relatively modest pipeline, a single step can consume as much as 10 minutes of execution time per image. With current multi-core CPUs, our current deployments have access to several hundred, even a relatively modest cluster easily has many 10s if not 100s of connections to an OMERO server. Our acquisition data, thanks to the availability of large format sCMOS cameras, reflects the current state of the art: 384 well, multi-field, 2048 x 2048 multi-channel datasets. A single plate acquisition of this type easily pushes 100GB.

In order to not repeatedly exhaust OMERO of resources or have the OMERO server require 10s of GBs of heap the management of OMERO stateful services, and by extension sessions, must be achieved.

It is here with respect to OMERO resource utilization, where @glencoesoftware has been focusing much of its attention, that the aforementioned trifecta of Bio-Formats, OmeroReader and OMERO compatibility with CellProfiler begins to show its design issues. This has not been helped by a bonafide OMERO bug in the Java client library which the OME team is hoping to include soon, possibly in a forthcoming 5.2.6 release:

• https://trello.com/c/NWT3qUma/172-client-del-session-still-alive

The longevity of OmeroReader is also a bit uncertain. The OME team is considering its deprecation and its possible future removal:

• have OmeroReader populate channel colors ome/openmicroscopy#4814 (comment)
• adjust how Java clients have Blitz close sessions ome/openmicroscopy#4817

Having to make an OMERO release to fix this bug is obviously less than ideal. As mentioned previously the OME team is attempting to clean up this particular mess.

Quite simply the current design of Bio-Formats reader interaction in CellProfiler, especially reader caching, when coupled with the naive way OmeroReader handles services and sessions is just not fit for purpose in an environment where headless, parallel execution on a cluster is commonplace. We have experimented with different execution modes for OmeroReader but the limitations of the Bio-Formats reader API are very real and troublesome. Furthermore, the limited way CellProfiler pipeline input can currently provide context information to downstream code is quite limiting. As a result, tight control of resources is quite difficult at present.

With all the above in mind we would like to propose re-opening this issue and continuing public discussion on the topic of pursuing use of OMERO’s Python bindings, at the very least as an option, to provide CellProfiler-OMERO integration. Providing an avenue to much improved integration moving forward. As mentioned previously, we are quite willing to pursue this work ourselves. Some topics for further discussion:

• Implications of transitioning CellProfiler-OMERO integration exclusively to pure Python
• CellProfiler team position on third-party Python modules and resultant deployment considerations for standalone end user as well as source deployments
• CellProfiler recent roadmap changes and proposed transition away from Bio-Formats

If this issue is not the right place for such a discussion and you would prefer to use your forum or get together on a video conference we’d be happy to do so.

Thanks for all your hard work! Let us know your thoughts.

/cc @emilroz @joshmoore @jburel @sbesson @0x00b1 @mcquin @AnneCarpenter

[mtbc]

the limitations of the Bio-Formats reader API are very real and troublesome

Are these documented in some way that can factor into design discussion for future breaking API changes?

[chris-allan]

We have taken some steps, via #2528, to clean up resources based on the current implementation. Furthermore, work has also started on a pure Python version of OmeroReader:

• https://github.com/glencoesoftware/omero-reader

This is for use with python-bioformats, via CellProfiler/python-bioformats#74, and by extension CellProfiler.

Much of this code has been in use for several weeks now with some fairly large workloads so we're fairly confident about its utility, at least in our hands, and welcome the community's feedback.

The problems referenced in this issue have also started to come up on the OME mailing lists. Relevant ome-devel mailing list thread:

• http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2016-December/003833.html

/cc @emilroz @joshmoore @jburel @sbesson @0x00b1 @mcquin @AnneCarpenter