Writing NIRS files to FIF is surprisingly slow

[rob-luke]

Describe the bug

I am reading NIRX NIRS files and writing them to fif files. This process takes longer than I expected.

It may be that this is how long writing takes. But I want to make sure there is nothing wrong with how we have written the read_raw_nirx function that is causing a slowdown.

Or perhaps fif is not the correct format I should be using? I selected fif as I know that it should work as its being tested here (called here).

Steps to reproduce

import os
import time
import mne

fnirs_data_folder = mne.datasets.fnirs_motor.data_path()
fnirs_raw_dir = os.path.join(fnirs_data_folder, 'Participant-1')
raw_intensity = mne.io.read_raw_nirx(fnirs_raw_dir, verbose=True, preload=False)

t = time.time()
raw_intensity.save("test_raw.fif")
print(time.time() - t)

On my PC this takes almost 2 minutes.

Expected results

I don't have an exact time I would expect this to take, but two minutes to process a 5 MB file seemed slow.

Actual results

About 2 mins.

Additional information

Platform:      Darwin-19.3.0-x86_64-i386-64bit
Python:        3.7.3 (default, Dec 13 2019, 19:58:14)  [Clang 11.0.0 (clang-1100.0.33.17)]
Executable:    /Library/Developer/CommandLineTools/usr/bin/python3
CPU:           i386: 16 cores
Memory:        32.0 GB
mne:           0.21.dev0
numpy:         1.18.2 {blas=openblas, lapack=openblas}
scipy:         1.4.1
matplotlib:    3.2.1 {backend=module://backend_interagg}
sklearn:       0.22.2.post1
numba:         0.48.0
nibabel:       3.0.2
cupy:          Not found
pandas:        1.0.3
dipy:          1.1.1
mayavi:        4.7.1
pyvista:       0.24.1
vtk:           8.1.2
PyQt5:         5.14.1

[agramfort]

is it also slow if you use preload=True? if not it's your lazy loading implementation that is slow. Certainly reads in more data then requested.

…

[rob-luke]

is it also slow if you use preload=True?

I was using preload=false as that's what's used in MNE-BIDS which I was starting to implement.

if not it's your lazy loading implementation that is slow.

Great pointer. I will look here for efficiency gains.

[larsoner]

IIRC during initial implementation we expected it to be slow. Looking at the implementation now, it actually reads every line until it finds the correct one(s), reads them, then breaks.

You could see if switching to the pandas implementation is faster. But maybe what would be even better (should be faster than pandas even) is to traverse the entire file exactly once during __init__ to learn where the line breaks (sample numbers) are, then when on demand reads are called do a fid.seek to the proper starting position and then read just the required number of lines.

[kylemath]

@kuziekj and I just moved the read_raw_boxy PR #7717 away from pandas

[larsoner]

@kuziekj and I just moved the read_raw_boxy PR #7717 away from pandas

Indeed it's better not to require it, at least to start. If there ends up being a performance bottleneck and we need to make an (optional!) requirement to read it to boost performance we can do that. But having the non-Pandas reader is still helpful.

[rob-luke]

Thank you all for the feedback and suggestions on where the code can be improved.

It seems the current implementation is correct but slow. I will leave this issue open as a reminder to address this in the future. But it does not seem urgent.

[G21R]

Good afternoon, I apologize for my poor English. Not so long ago I started working in a laboratory where fNIRS data is analyzed.
The question is this. I processed raw data, did filtering, deleted bad channels. Please tell me how I can save the result now - in the future, these processed data will be sent back to NIRS Sport. Can I use the save() method to save this. So that the equipment then correctly reads the processed data.
My idea is to use .save("data.fif") will it work?