Problem: Reading the high-resolution layers of the pyramid image for registration at a lower resolution level.

[juanenofbit]

Hi Chandler,

After several months without using it, I have updated valis to the latest version (1.0.0.rc15) and relaunched a script I had prepared for registration of four czi images (to test if with the update I got better or different results)

When I used the old 1.0.0.rc11 release, the reading of images for the registration process was fast because it went directly to some low resolution layer (according to the size chosen in the parameters). Only the high resolution layer was read in the final warping process registrar.warp_and_save_slides

Now, after this command:
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

The following message appears, about SKF4J, before reading the high-resolution pyramid:

==== Converting images

  0%|                                                                                                             | 0/4 [00:00<?, ?it/s]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

JVM has been initialized. Be sure to call registration.kill_jvm() or slide_io.kill_jvm() at the end of your script.

  0%|                                                                                                         | 0/11760 [00:00<?, ?it/s]
  1%|█                                                                                              | 126/11760 [00:07<11:56, 16.23it/s]
  2%|█▌                                                                                             | 189/11760 [00:13<12:39, 15.23it/s]
  2%|█▌                                                                                             | 191/11760 [00:17<17:28, 11.03it/s]
  2%|█▌                                                                                             | 192/11760 [00:19<27:10,  7.10it/s]

And so on until 100% of all images (15 minutes per image, which are very large, before recording in low resolution).

When updating to the 1.0.0.rc15, I installed maven according to the documentation, also updating bioformats-jar to 2020.5.27 version.

Best,
Juan

[cdgatenbee]

Hi @juanenofbit,

Good to hear from you again. I think what might be happening is that valis is using the slide_io.CziJpgxrReader reader, which is a "fallback" slide reader in the event that BioFormats (via the slide_io.BioFormatsSlideReader) runs into errors reading czi images that use JPEG-XR compression. Unfortunately, that reader does require building the entire full resolution image and then resizing it. But could you confirm this by seeing which type of SlideReader you get when you call slide_io.get_slide_reader(src_f=path_to_slide)? Also, you could try forcing valis to use BioFormats by setting reader_cls=slide_io.BioFormatsSlideReader when you call Valis.register, which (if it works) should have similar read times as before. Please let me know how it goes.

Best,
-Chandler

[juanenofbit]

Hi Chandler, thank you for your prompt reply!

I have cheched which reader is being used by default: reader_cls = slide_io.get_slide_reader(src_f=reference_image), which is inneficient (~15 minutes), and returns a valis.slide_io.BioFormatsSlideReader

But when I explicitly force the BioFormats reader, the algorithm efficiently reads low-resolution images, as before.

rigid_registrar, non_rigid_registrar, error_df = registrar.register(reader_cls=slide_io.BioFormatsSlideReader)

So this works although something strange happens with the bioformats reader.

Regards,
Juan

[cdgatenbee]

Hi @juanenofbit,
Thanks for the update! Glad you were able to get the read speeds back to where they were. I'll look into the BioFormats reader to figure out what's causing the unexpected behavior.

Best,
-Chandler

[juanenofbit]

Hi Chandler,

I am having the same problem (reading the image at high resolution level, taking a lot of time) when I want to save registered images at a lower resolution (for example using level=3)

registrar.warp_and_save_slides(dir_output, level=3, non_rigid=True, compression="lzw", crop="reference")

How can I specify the slide_io.BioFormatsSlideReader in this case?

There's another concern related to image compression. It seems that only the "lzw" compression format produces compatible images with QPath, and when I attempt to use "jpeg" or "jp2k" compression formats, the resulting images are not readable, at least within QuPath. Each "lzw" compressed image takes up approximately 20GB of space. I'd like to use the lossy compressions, as it should significantly reduce the file size to about one-tenth of what "lzw" produces, (around 2GB)

Thansk!
Juan

[cdgatenbee]

Hi @juanenofbit,
Do you think you could share one of your czi images and I can use to see what's going on the slide_io.BioFormatsSlideReader? I can also use that image to test saving with "jpeg" or "jp2k" compression and opening in QuPath. Regarding the compression method and file size, I think it should be fairly straightforward to update the code to allow for lossy compression, so I'll go ahead and add that too.

Best,
-Chandler

[juanenofbit]

Thank you , Chandler

I send you the download link to a CZI image by e-mail

[cdgatenbee]

Got it, thanks @juanenofbit! I'll keep you updated.

-Chandler

[cdgatenbee]

Hi @juanenofbit,
It seems the reason the BioformatsReader isn't working is related to the .czi images being compressed using JPEGXR, and bioformats not being able to find a suitable jxrlib(see here, here, and here for similar issues). Perhaps one of those solutions will work for you? Personally, I have the same issue on my apple silicon mac (even with the Ubuntu based Docker image), despite having jxrlib installed, which is why I created the CziJpgxrReader. Unfortunately, aicspylibczi (which CziJpgxrReader is based on) throws a MemoryError: std::bad_alloc error when trying to open a scaled version of the slide, and so valis has to construct the whole image and then resize size it, which is time consuming. Anyway, I'm still working on adding the options for lossy compression to reduce file size, but I should have that complete for the next update.

Best,
-Chandler

[juanenofbit]

Thank you, Chandler
seems to be a complicated problem. I will wait for the next update to check the lossy compression options.

[cdgatenbee]

Hi @juanenofbit,
Just wanted to let you know that the most recent update (1.0.1) includes lossy jpeg and jpeg2000 compression. When using functions related to saving slides, just set the Q parameter to something less than 100, and compression to "jpeg" or "jp2k" and your file sizes should be much smaller. I also verified that compressed RGB images appear as expected when opening with QuPath, which had been an issue before. Unfortunately, I still don't have a good solution for quickly reading czi that have jpegxr compression, but I'll keep my eyes open for something.

Best,
-Chandler

[juanenofbit]

Hi Chandler, Thank you for the upgrade!

When I try to upgrade using pip install valis-wsi --upgrade, in a conda envornment, the following message appears:

Requirement already satisfied: valis-wsi in ./anaconda3/envs/valis/lib/python3.9/site-packages (1.0.0rc15)
Collecting valis-wsi
  Obtaining dependency information for valis-wsi from https://files.pythonhosted.org/packages/db/91/c2431a4e07b365a5025fbd7b7adcce17fd9fdcf0aa5196cb30b75252e261/valis_wsi-1.0.1-py3-none-any.whl.metadata
  Downloading valis_wsi-1.0.1-py3-none-any.whl.metadata (7.9 kB)
ERROR: Packages installed from PyPI cannot depend on packages which are not also hosted on PyPI.
valis-wsi depends on aicspylibczi@ git+https://github.com/AllenCellModeling/aicspylibczi.git

I have also tried to install valis in a new clean environment with the same error

Best,
Juan

[cdgatenbee]

Hi @juanenofbit,
Thanks for brining this to my attention. I think I know the solution and will try to get it fixed asap. I'll let you know when it looks like it's been fixed.

Best,
-Chandler

[cdgatenbee]

Hi @juanenofbit,
It looks like PyPi and TestPyPi are under maintenance right now, and so I cant upload the package yet. In the meantime, maybe you could try using pip to install the most recent version directly from github?

pip install git+https://github.com/MathOnco/valis.git

Hoping to have this fixed on PyPi sometime today.

Best,
-Chandler

[cdgatenbee]

Hi @juanenofbit,
I re-uploaded the package to PyPi (now version 1.0.2), and the normal installation via pip seems to be working again. Let me know if you run into any issues though.

Best,
-Chandler