Suggestion for RCA memory issue

[aguinot]

Here are some suggestions that might improve the memory management :
rca_runner.py

• At line 385:

starcat = fits.open(starcat_path)

I would replace by:

starcat =  fits.open(starcat_path, memmap=False)
positions = starcat[2].data['GLOB_POSITION_IMG_LIST']
stars = starcat[2].data['VIGNET_LIST']
ccds = starcat[2].data['CCD_ID_LIST']
masks = starcat[2].data['MASK_LIST']
starcat.close()

Then you can put those parameters into a dictionary for example and feed that to your fonction mccd_rca_fit. I don't really like the idea of feed an instanced class for a fits catalog..

I would do something similar for mccd_validation. Here is my suggestion

testcat = fits.open(test_path, mmemap=False)

# Saving file dictionary
star_dict = {}

# Preparing data in ccd-list format
# Loading
my_ccd_list = testcat[2].data['CCD_ID_LIST'].astype(int)
ccd_unique_list = np.unique(testcat[2].data['CCD_ID_LIST']).astype(int)

positions = testcat[2].data['GLOB_POSITION_IMG_LIST']
stars = testcat[2].data['VIGNET_LIST']
masks = testcat[2].data['MASK_LIST']
ccds = testcat[2].data['CCD_ID_LIST']
try:
    RA_pos = testcat[2].data['RA_LIST']
    DEC_pos  =testcat[2].data['DEC_LIST']
except Exception:
    pass
testcat.close()

[aguinot]

An other thing that we can consider is for the module that merge all the ccd in one, we could try using the sqlitedict library instead of fits file (or numpy, or vaex).
Also you could remove all the column that you don't need (it might already be the case).

[tobias-liaudat]

This has been solved with the PR #396 .

It had to do with memmap behaviour when opening fits files.