Deviating values while reading data from EDF and BrainVision formats

[MaxvandenBoom]

Hi!

I am currently building a BIDS Docker App that takes three iEEG data formats: MEF3, EDF and BrainVision. In order to read the EDF and Brainvision formats I am using MNE-tools (mne-python, latest stable version 0.19.2) in Python 3.8.

To test the reading of data in Python I converted one dataset into all three formats. While being successful at loading each of the data formats into python, the raw data values coming from MNE seem to differ (see the attached image below):

• The raw values that MNE returns from the Brainvision format seem to be the correct values but scaled down
• The raw values that MNE returns from the EDF format are partial correct, but are wrong on values below 0.

The mne.io.read_raw_edf and mne.io.read_raw_brainvision functions were used to read the data and I retrieved the raw values using the methods described here:
https://mne.tools/dev/auto_tutorials/raw/plot_10_raw_overview.html#extracting-data-from-raw-objects

In order to see if the cause was not in the data files, I tried two other methods to read the data:

• I read the exact same data files in Matlab using the Fieldtrip package and there the BrainVision and EDF did show the correct values.
• I used another Python EDF reader (PyEdfLib), which also gave the correct values.

These led me to believe it is something MNE reader, rather than something in the data.

I am not sure whether there is a conversion or data casting step in the reading of the raw data that I am missing? Or whether this could be a bug in the MNE reading function?
@jasmainak @dorahermes

Hope you can help. If more information is needed, then please let me know...

Thanks in advance,
Max

---

Attached, results using different readers:

[agramfort]

@MaxvandenBoom can you share an EDF file privately so we can debug this. Could be a silly bug in Edf reader.

[drammock]

@MaxvandenBoom to me it looks like the BrainVision format is OK, it's just off by a factor of 1e6. MNE-Python always converts EEG data into Volts when reading, even if the value in the data container was in μV. The EDF values does look like a bug though.

[drammock]

(for more info about unit conversion in MNE-Python, see here: https://mne.tools/dev/overview/implementation.html#internal-representation-units)

[jasmainak]

@MaxvandenBoom can you explain how you converted your data to the three file formats? What format was the data originally in?

[cbrnr]

Interesting, I just wanted to add that the EDF values differ from the MEF3 and BrainVision even with PyEDFlib and FieldTrip. It looks like something funky going on either with the data type and/or with the scaling.

Either way, as others have already noted, we need all files as well as details on how you created these files.

[MaxvandenBoom]

@agramfort Yes definitely, I will prepare a dataset
@drammock Thank you, that indeed explains the BrainVision values.
@jasmainak The EDF and BrainVision formats were derived from MEF3 format, conversion was done using Fieldtrip
@cbrnr Indeed, I have noticed that as well, but wanted to focus on the obvious incorrect values. It would be nice to find an explanation for those fluctuations as well :)

I will supply some datasets and the conversion scripts I used in one go

[dorahermes]

For BrainVision: Thank you guys for resolving the BrainVision difference, this is super helpful!

For EDF: I wrote these files to EDF and BrainVision using Fieldtrip, but there seems already a difference between the written .edf and the original data, even if loaded in Fieldtrip, so this is not due to MNE Python (these data might actually be >16 bit, I'll check on our end).

General question: Does anyone know of an already existing standard dataset that exists in EDF and BrainVision that we could test for the Docker/BIDS App? It would be great to have an example testset for BIDS app development, we'll try to make one if not.

[agramfort]

we have some tiny date used for testing here: https://github.com/mne-tools/mne-testing-data/tree/master/ <https://github.com/mne-tools/mne-testing-data/tree/master/EDF> mne has tests for IO on all these files. What would the app do?

[dorahermes]

That's awesome, thank you!

The app will do an automated detection of early onsets in cortico-cortical evoked potential (CCEP) data. @MaxvandenBoom is making sure first that all the BIDS data formats can be read correctly and the units make sense.

[agramfort]

great can you share the repo of where you plan to put this code? you'll be using brainlife.io ?

…

[dorahermes]

It's still under development, but we'll be putting the N1 detection on my lab's gitHub site:
https://github.com/MultimodalNeuroimagingLab

[agramfort]

thanks for sharing. Let us know how it goes and if we can help in some way.

…

[MaxvandenBoom]

@agramfort @dorahermes

I tried the BrainVision and EDF dataset you suggested from https://github.com/mne-tools/mne-testing-data/tree/master/

• The BrainVision data was read correctly given drammock's explanation
• The EDF data has the same bug with negative values. I read the file \EDF\test_edf_overlapping_annotations.edf using MNE, pyEDFlib and Fieldtrip. The results are shown in the attached image below.

This is the code I used in Python:

from mne.io import read_raw_edf, read_raw_brainvision
from pyedflib import EdfReader

# read the edf data using MNE
mne_edf = read_raw_edf('D:\mne-testing-data-master\mne-testing-data-master\EDF\\test_edf_overlapping_annotations.edf', eog=None, misc=None, stim_channel=False, preload=True, verbose=None)
mne_vals = mne_edf[0, 0:20][0]

# read the edf data using pyedflib
lib_edf = EdfReader('D:\mne-testing-data-master\mne-testing-data-master\EDF\\test_edf_overlapping_annotations.edf')
lib_vals = lib_edf.readSignal(0)[0:20]

This is the code I used in Matlab:

edf_data = ft_read_data('D:\mne-testing-data-master\mne-testing-data-master\EDF\test_edf_overlapping_annotations.edf');
edf_vals = edf_data(1, 1:20)

So I think you were right and it might be some silly bug in the MNE edf reader... Also, should the EDF internal values be in Volts?

---

[larsoner]

This appears to be due to doing stim_channel=False, which gets interpreted as stim_channel=0, i.e., the first channel is treated as STIM and hence non-negative/unsigned.

[larsoner]

... which is because of this:

 isinstance(False, int)

Note that this is not documented behavior (bool is not in the docstring of accepted values), but we can at least add some aliases to avoid this.