Reduce reader memory consumption#228
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #228 +/- ##
==========================================
- Coverage 45.50% 45.02% -0.48%
==========================================
Files 22 22
Lines 2149 2183 +34
==========================================
+ Hits 978 983 +5
- Misses 1171 1200 +29
|
|
Thanks for the PR @marcovarrone, I will check it and get back to you. |
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
feat: improve cosmx loading RAM performances
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
|
Closing because it's a bit outdated and dask already uses chunking to adapt the amount of memory required. I will create smaller and separate pull requests for each reader with appropriate credits for the nice work done by @laudmt, @cgeyskens and @frankandreace during the scverse x Owkin hackathon. |
|
Thanks @marcovarrone for the update, this went out of our radar. I just add a chat with @melonora regarding this and saw the updates. I will follow up on the Xenium PR. |
It adds a new
output_pathparameter in a reader's function that allows to save every element of the spatialdata object as soon as it's created.This frees up part of the memory during the function's execution instead of maintaining the whole spatialdata object in memory and then saving it with
sdata.write.I've done it only for Xenium but if I receive the ok I can implement it for the rest of the readers.