Skip to content

Conversation

@larsoner
Copy link
Member

@rob-luke some issues so far:

  1. There is Wave[nm],695,830 in the header, but also Wave Length,CH1(700.1),CH1(829.1),CH2(699.3),CH2(827.9),.... I forced the 695-ish ones to be 695, and 830-ish ones to be 830. Is this the right thing to do?
  2. I don't see any explicit source-detector pairings in the file, so I just named them consecutively S1_D1, S2_D2, etc. I thought about trying to construct this based on the wavelengths in the channel names, but there was at least one pairing that was identical in my file so that seemed like a no-go.
  3. No idea what the montage is. (Might be related to (2).)
  4. Don't know what Exception Ch,0,0,0,0,0,0,1,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0 means (should end up in info['bads'], maybe?)

I can bother the folks who got me these data to get some of this information, but figured I'd check with you to see if you knew already.

Closes #9355

@rob-luke
Copy link
Member

rob-luke commented May 12, 2021

I'm not 100% sure on the answer to your questions, but we can try and puzzle it out together.

There is Wave[nm],695,830 in the header, but also Wave Length,CH1(700.1)

Could Wave be the equivalent of SNIRF-ProbeWavelength and WaveLength be the equivalent of SNIRF-WavelengthActual. If so, I would suggest we use Wave (the rounded values) in the channel name field, and store the precise wavelength in the info field (and use the precise value for the beer lambert calculation (currently we assume same wavelength for each pair, but would only be a line or two change to support different wavelength for each channel - although i think the affect of this on output would be negligible and other toolboxes don’t do it)).

Location for precise wavelength could be something like

info['chs'][ch_idx2 * 2 + 1]['loc'][9] = fnirs_wavelengths[1]

I don't see any explicit source-detector pairings in the file

It's entirely possible that these are not stored in the files you were provided, and are stored separately or expected to be known. For example, some vendors have just a few preset montages, if this is the case we could store a few preset montages (like we did with the Artinis devices #9141). Or, it could be that the vendor allows the optodes to be placed in any position, and does not provide a standard way to store the locations and leaves that to the user. In which case we could ask users to write an elc file. But this is all speculation, do you have a copy of the vendor manual or can you ask your collaborator how they unusually specify optode positions?

I think you are correct that point 2 is same as 3.

Don't know what Exception Ch,0,0,0,0,0,0,1,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0 means

No idea, but I think it many be bads like you suggest. Is it the same in all files?

Does the vendor provide a standard set of files on their website? Just to ensure you actually were given a complete recording dataset (sometimes collaborators forget to send you parts of a recording and maybe they left out the montage file).

@rob-luke
Copy link
Member

I also have some Hitachi files I could test locally with.

@larsoner
Copy link
Member Author

Okay I think it works testing locally with:

raw = mne.io.read_raw_hitachi(fname)

# montage
need = sum(([f'S{n}', f'D{n}'] for n in range(1, 8)), []) + ['S8']
have = 'F7 F5 FC5 F3 FC3 FT7 T7 C5 C3 TP7 CP5 CP3 P7 P5 P3'.split()
mon = mne.channels.make_standard_montage('standard_1020')
mon.rename_channels(dict(zip(have, need)))
raw.set_montage(mon)
fig = mne.viz.plot_alignment(
    subject='fsaverage', trans='fsaverage', info=raw.info,
    surfaces=dict(pial=0.2, head=0.2), coord_frame='mri', dig=True,
    fnirs=['channels', 'pairs', 'sources', 'detectors'],
    show_axes=True, mri_fiducials=True)

My ordering of the have is clearly wrong (I need to reorderd have based on what my collaborator tells me) but it otherwise works:

Screenshot from 2021-05-14 13-24-18

If I use the 3x5.pos from the HOMER Hitachi conversion routines, which is just some random digitization, order is correct (but locations are wrong, of course):

Screenshot from 2021-05-14 13-23-15

Once I have the correct order in the docstring example this should be good to go, so feel free to test on your data.

@larsoner larsoner changed the title WIP: Add support for Hitachi fNIRS MRG, ENH: Add support for Hitachi fNIRS May 14, 2021
@larsoner
Copy link
Member Author

I fixed the order in the docstring so this is good to go from my end:

Screenshot from 2021-05-14 15-59-34

@rob-luke
Copy link
Member

Sorry its taken me a few days to get to this @larsoner. I have run some of my data through your reader, and unfortunately it is not reading the data. Your reader works with Hitachi files with file version 1.18, but fails on data with file version 1.25. Whats the best way to proceed here? Add a check and say we only support old file versions? Or expand support to the new file format too?

As these are just text files with a custom header I will post some file info I have below. I can also share some data with you privately if that helps?

@rob-luke
Copy link
Member

Here is an example file (with me having anonymised various aspects of the metadata)...

Header
File Version,1.25
Patient Information
ID,REDACTED
Name,
Comment,sentence,,
Birth Date,1999/09/09
Age, 99y
Sex,Male
Analyze Information
AnalyzeMode,Continuous
Pre Time[s],9.0
Post Time[s],7.0
Recovery Time[s],12.0
Base Time[s],5
Fitting Degree,1
HPF[Hz],No Filter
LPF[Hz],No Filter
Moving Average[s],0.1
Measure Information
Date,2020/02/02 11:20
Probe Type,adult
Mode,3x11
Wave[nm],695,830
Wave Length,CH1(703.6),CH1(829.0),CH2(703.9),CH2(829.3),CH3(703.9),CH3(829.3),CH4(703.9),CH4(828.8),CH5(703.9),CH5(828.8),CH6(703.1),CH6(828.8),CH7(703.1),CH7(828.8),CH8(702.9),CH8(829.0),CH9(702.9),CH9(829.0),CH10(703.6),CH10(829.0),CH11(703.6),CH11(829.0),CH12(703.4),CH12(829.0),CH13(703.9),CH13(829.3),CH14(703.6),CH14(828.5),CH15(703.9),CH15(828.8),CH16(704.1),CH16(829.8),CH17(703.1),CH17(828.8),CH18(704.4),CH18(829.0),CH19(702.9),CH19(829.0),CH20(703.9),CH20(828.8),CH21(703.6),CH21(829.0),CH22(703.4),CH22(829.0),CH23(703.4),CH23(829.0),CH24(703.6),CH24(828.5),CH25(703.6),CH25(828.5),CH26(704.1),CH26(829.8),CH27(704.1),CH27(829.8),CH28(704.4),CH28(829.0),CH29(704.4),CH29(829.0),CH30(703.9),CH30(828.8),CH31(703.9),CH31(828.8),CH32(704.1),CH32(828.8),CH33(703.4),CH33(829.0),CH34(703.1),CH34(828.8),CH35(703.6),CH35(828.5),CH36(704.4),CH36(828.8),CH37(704.1),CH37(829.8),CH38(703.6),CH38(828.5),CH39(704.4),CH39(829.0),CH40(703.9),CH40(829.0),CH41(703.9),CH41(828.8),CH42(704.4),CH42(829.0),CH43(704.1),CH43(828.8),CH44(703.1),CH44(828.8),CH45(703.1),CH45(828.8),CH46(704.4),CH46(828.8),CH47(704.4),CH47(828.8),CH48(703.6),CH48(828.5),CH49(703.6),CH49(828.5),CH50(703.9),CH50(829.0),CH51(703.9),CH51(829.0),CH52(704.4),CH52(829.0)
Analog Gain,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,6.58823500,6.58823500,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,6.58823500,6.58823500,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,6.58823500,6.58823500,2.74509800,2.74509800,1.25490200,1.25490200,4.07843100,4.07843100,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,2.74509800,2.74509800,2.74509800,2.74509800,4.07843100,4.07843100,4.07843100,4.07843100,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200,1.25490200
Digital Gain,5.99000000,3.61000000,1.93000000,1.12000000,5.43000000,3.20000000,7.11000000,4.02000000,11.82000000,6.06000000,16.42000000,7.48000000,15.82000000,7.56000000,12.13000000,6.80000000,14.62000000,7.78000000,5.35000000,2.57000000,10.27000000,5.01000000,52.84000000,26.95000000,9.75000000,6.12000000,6.97000000,3.99000000,11.31000000,5.91000000,2.26000000,1.15000000,9.49000000,4.58000000,3.21000000,2.25000000,2.97000000,1.65000000,12.34000000,6.77000000,33.81000000,17.23000000,7.10000000,2.92000000,12.78000000,7.02000000,17.47000000,9.94000000,16.93000000,9.55000000,6.87000000,3.47000000,5.96000000,3.07000000,20.56000000,13.42000000,3.67000000,2.53000000,17.39000000,8.72000000,2.96000000,1.61000000,14.25000000,5.67000000,8.87000000,4.21000000,21.68000000,11.89000000,12.24000000,6.47000000,28.07000000,16.60000000,13.95000000,7.05000000,9.37000000,5.53000000,18.15000000,11.62000000,39.72000000,19.87000000,20.78000000,9.56000000,91.12000000,43.86000000,17.86000000,7.84000000,5.47000000,3.40000000,4.83000000,3.38000000,7.61000000,5.15000000,4.65000000,2.86000000,1.33000000,0.88000000,16.06000000,11.46000000,11.94000000,6.47000000,25.92000000,12.83000000,4.35000000,2.44000000
Sampling Period[s],0.1
StimType,STIM
Stim Time[s]
A,15,B,15,C,15,D,15,E,15,F,15,G,15,H,15,I,15,J,15
Repeat Count,20
Exception Ch,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0






Data
Probe1,CH1(703.6),CH1(829.0),CH2(703.9),CH2(829.3),CH3(703.9),CH3(829.3),CH4(703.9),CH4(828.8),CH5(703.9),CH5(828.8),CH6(703.1),CH6(828.8),CH7(703.1),CH7(828.8),CH8(702.9),CH8(829.0),CH9(702.9),CH9(829.0),CH10(703.6),CH10(829.0),CH11(703.6),CH11(829.0),CH12(703.4),CH12(829.0),CH13(703.9),CH13(829.3),CH14(703.6),CH14(828.5),CH15(703.9),CH15(828.8),CH16(704.1),CH16(829.8),CH17(703.1),CH17(828.8),CH18(704.4),CH18(829.0),CH19(702.9),CH19(829.0),CH20(703.9),CH20(828.8),CH21(703.6),CH21(829.0),CH22(703.4),CH22(829.0),CH23(703.4),CH23(829.0),CH24(703.6),CH24(828.5),CH25(703.6),CH25(828.5),CH26(704.1),CH26(829.8),CH27(704.1),CH27(829.8),CH28(704.4),CH28(829.0),CH29(704.4),CH29(829.0),CH30(703.9),CH30(828.8),CH31(703.9),CH31(828.8),CH32(704.1),CH32(828.8),CH33(703.4),CH33(829.0),CH34(703.1),CH34(828.8),CH35(703.6),CH35(828.5),CH36(704.4),CH36(828.8),CH37(704.1),CH37(829.8),CH38(703.6),CH38(828.5),CH39(704.4),CH39(829.0),CH40(703.9),CH40(829.0),CH41(703.9),CH41(828.8),CH42(704.4),CH42(829.0),CH43(704.1),CH43(828.8),CH44(703.1),CH44(828.8),CH45(703.1),CH45(828.8),CH46(704.4),CH46(828.8),CH47(704.4),CH47(828.8),CH48(703.6),CH48(828.5),CH49(703.6),CH49(828.5),CH50(703.9),CH50(829.0),CH51(703.9),CH51(829.0),CH52(704.4),CH52(829.0),Mark,Time,BodyMovement,RemovalMark,PreScan
1,2.01614380,2.01263428,2.01354980,2.01248169,2.02392578,2.01644897,2.02453613,2.01934814,2.01873779,2.01919556,2.02087402,2.01889038,2.02621460,2.03781128,2.02423096,2.02011108,2.00881958,2.00637817,2.02087402,2.01644897,2.02209473,2.00988770,2.03155518,2.01919556,2.02148438,2.01370239,2.01324463,2.00973511,2.00973511,2.01477051,2.02117920,2.02636719,2.00805664,2.01583862,2.02529907,2.03033447,2.00424194,1.99935913,2.01202393,2.01568604,2.01583862,2.01095581,2.02545166,2.01721191,2.02758789,2.02957153,2.00622559,2.00073242,2.00759888,2.00485229,2.02041626,2.03094482,2.01782227,2.01675415,2.00820923,2.01110840,2.01354980,2.00637817,2.01538086,2.00271606,2.01156616,2.01217651,2.01736450,1.99386597,2.03353882,2.02362061,2.03598022,2.02728271,2.00866699,1.99676514,2.00714111,2.00744629,2.02087402,2.01492310,2.01202393,2.01644897,2.00057983,1.99584961,2.01370239,1.99050903,1.99783325,1.99371338,2.01431274,1.99920654,1.99630737,1.97814941,2.02545166,2.01126099,2.02102661,2.00790405,2.00912476,1.99813843,2.01156616,1.99981689,1.99829102,1.99523926,2.01141357,2.00424194,2.00164795,1.98730469,2.03948975,2.00637817,2.00958252,1.99615479,0,15:40:49.15,0,0,1
2,2.02087402,2.02194214,2.01782227,2.02026367,2.02743530,2.02453613,2.02789307,2.02758789,2.02255249,2.02880859,2.02819824,2.02880859,2.03445435,2.04879761,2.02850342,2.02865601,2.01583862,2.01477051,2.02529907,2.02499390,2.03033447,2.02178955,2.03308105,2.02636719,2.02545166,2.02224731,2.01690674,2.01751709,2.01522827,2.02423096,2.02590942,2.03536987,2.01507568,2.02850342,2.03033447,2.04040527,2.00729370,2.00546265,2.01431274,2.02407837,2.02224731,2.02346802,2.02972412,2.02682495,2.03186035,2.03720093,2.01171875,2.00851440,2.00897217,2.01370239,2.02468872,2.04177856,2.02224731,2.02606201,2.01339722,2.02499390,2.01766968,2.01583862,2.01690674,2.00851440,2.01568604,2.02117920,2.02087402,2.00164795,2.03689575,2.03247070,2.04132080,2.03613281,2.01110840,2.00332642,2.01293945,2.01583862,2.02560425,2.02224731,2.01828003,2.02621460,2.00637817,2.00607300,2.01431274,1.99890137,2.00714111,1.99813843,2.01721191,2.00942993,1.99874878,1.98593140,2.02941895,2.02087402,2.02743530,2.01690674,2.01263428,2.00668335,2.01690674,2.00592041,2.00210571,2.00210571,2.01568604,2.01110840,2.00378418,1.99447632,2.04177856,2.01278687,2.01217651,2.00210571,0,15:40:49.26,0,0,1
3,2.02545166,2.03063965,2.02102661,2.02774048,2.03079224,2.03170776,2.03170776,2.03521729,

... and so on and so forth

@larsoner larsoner changed the title MRG, ENH: Add support for Hitachi fNIRS WIP, ENH: Add support for Hitachi fNIRS May 17, 2021
@larsoner
Copy link
Member Author

@rob-luke I fixed some bugs and made the code more general. There is a bit of a problem, though, in that the nirs-toolbox code for getting the src-det pairs produces a result for example for 3x5:

>> probe.link

ans =

  44×3 table

    source    detector    type
    ______    ________    ____

      1          1        695 
      1          1        695 
      2          1        830 
      2          1        830 
      4          1        830 
      4          1        830 
      2          2        695 
      2          2        695 

That both seems wrong -- these I think/assume are ordered the same way as the channels, but they have the first two channels with the same frequency for source and detector -- and does not match with the pairing in Hitachi 2 Homer:

    SD.MeasList(1,:) =  [1 1 1 1];
    SD.MeasList(2,:) =  [1 1 1 2];
    SD.MeasList(3,:) =  [2 1 1 1];
    SD.MeasList(4,:) =  [2 1 1 2];
    SD.MeasList(5,:) =  [2 2 1 1];
    SD.MeasList(6,:) =  [2 2 1 2];
    SD.MeasList(7,:) =  [1 3 1 1];
    SD.MeasList(8,:) =  [1 3 1 2];

So I'm hesitant to use this to get your 3x11 pairings, and 3x11 is missing from Hitachi 2 Homer. Any idea where we can get the pairings?

@larsoner
Copy link
Member Author

@rob-luke I think I figured out the pattern that gave rise to the Hitachi 2 Homer pairings, and I coded it and test it. Assuming it's correct and the iteration on the 3x11 is the same as the 3x5 (and not, say, column-wise instead of row-wise) then this should work.

@rob-luke
Copy link
Member

Even with the modification above I am still getting an error with my files. I will email you a file to test with.

the error I am getting is

  File "/Volumes/MassStorage2TB/rluke/Repositories/mne-python/mne/io/nirx/nirx.py", line 424, in _read_csv_rows_cols
    x.shape = (stop - start, -1)
ValueError: cannot reshape array of size 2750 into shape (26,newaxis)

@larsoner larsoner changed the title WIP, ENH: Add support for Hitachi fNIRS MRG, ENH: Add support for Hitachi fNIRS May 18, 2021
@larsoner
Copy link
Member Author

Okay looks like it was a line-endings problem, should be fixed now @rob-luke

Copy link
Member

@rob-luke rob-luke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on a few local files and they read and return the correct data type and metadata. But I don't have access to a machine myself, so cant do detailed testing. From what I can test it looks good to me.

@larsoner
Copy link
Member Author

A collaborator had details of the electrode montage and what I get from MNE now matches their output, so I think we're good to merge

@larsoner larsoner merged commit 5bc18f9 into mne-tools:main May 18, 2021
@larsoner larsoner deleted the hitachi branch May 18, 2021 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENH: Add data reader for Hitachi NIRS files

2 participants