Add more referencing options #1670

wmvanvliet · 2014-11-25T14:14:54Z

This adds a base function apply_reference that can apply various
different referencing schemes. Two convenience functions are also
added that provide a convenient interface to the base function:

set_eeg_reference: apply a simple reference to all EEG channels
set_bipolar_reference: apply a bipolar reference (or more)

A helper function _parse_channel_list is also added to help functions
be more flexible in how they allow channels to be specified.

dengemann · 2014-11-25T15:21:13Z

mne/channels/channels.py

@wmvanvliet which problem does this solve?

coveralls · 2014-11-26T08:22:45Z

Coverage increased (+31.39%) when pulling df8b9e6 on wmvanvliet:reference into 4abb240 on mne-tools:master.

agramfort · 2014-11-26T08:34:23Z

mne/io/base.py

agramfort · 2014-11-26T08:41:21Z

should we have a way to store in info or somewhere what type of referencing was applied?

How about a method to revert a reference. I am thinking about this for inverse modeling as you don't want anything else than average ref for inverse modeling. This undo_reference or revert_reference method should be able to be applied to any data container.

wmvanvliet · 2014-11-26T09:56:14Z

It would be very hard to invert referencing and apply CAR retroactively. For example, channels may be dropped along the way. If the user wants to perform source modelling, they should definitely stick to a CAR reference, which is and should be the default. The user has to explicitly define another reference and presumably the user knows what he/she is doing.

agramfort · 2014-11-26T10:26:53Z

The user has to explicitly define another reference and presumably the user knows what he/she is doing.

you're optimistic :)

Then we should have a flag that says if data has been referenced is a
non-compatible way with inverse modeling. Then raise an error.

dengemann · 2014-11-26T10:29:01Z

Then we should have a flag that says if data has been referenced is a
non-compatible way with inverse modeling. Then raise an error.

+1

wmvanvliet · 2014-11-26T10:41:13Z

I can add the flag, but the check should be implemented by someone familiar with the source modelling code.

agramfort · 2014-11-26T10:48:25Z

sounds like a plan

dengemann · 2014-11-26T10:55:00Z

fair enough

wmvanvliet · 2014-11-26T11:01:32Z

Shall I add a constant to io.constants.FIFF for some referencing types (including OTHER)? Then source modelling code can check assert info['reference'] == FIFF.FIFFV_REF_CAR

agramfort · 2014-11-26T17:44:22Z

how many of such options would we add? other option is have a boolean flag.

wmvanvliet · 2014-11-26T17:47:35Z

but we must now what kind of reference was applied, CAR or something else…
The flag would become ‘car_reference_applied’ or something similar.

On 26 Nov 2014, at 19:44, Alexandre Gramfort notifications@github.com wrote:

how many of such options would we add? other option is have a boolean flag.
—
Reply to this email directly or view it on GitHub #1670 (comment).

Marijn van Vliet
w.m.vanvliet@gmail.com

agramfort · 2014-11-26T21:33:58Z

but we must now what kind of reference was applied, CAR or something else…
The flag would become ‘car_reference_applied’ or something similar.

the thing is that CAR works with an SSP / proj and it's nice and elegant.

I feel we should stick to it when we apply CAR (ie. create proj and apply it)

For all the other reference options set a flag that says that it will
not be compatible with inverse modeling.

wdyt?

wmvanvliet · 2014-11-28T08:05:41Z

Agreed.

Lets recap the EEG referencing arguments so far:

The default reference should be CAR
It should be straightforward for the user to apply another reference
When another reference is set, source localization will be unreliable/wrong
Computing the CAR on bad channels is bad

1. The default reference should be CAR

Upon loading a raw file, a CAR projector is automatically added for the EEG. Any bad channels already marked in the raw data are excluded.

2. It should be straightforward for the user to apply another reference

The apply_reference function applies a different reference, but not through an SSP projector (this is as far as I know only possible for CAR). The convenience functions set_eeg_reference and set_bipolar_reference wrap this function to provide a nice interface to the user.

If a CAR SSP projection is present in the data, it should be removed by this function. Otherwise things become very confusing for the user. When an instance of Epochs is created, the current behaviour is to add a CAR projection if not present. Instead, the flag should be checked and respected. Also, the user should be able to use the set_eeg_reference to specify the data has already been referenced (some systems, like the Emotiv EPOC, do referencing at the driver level) and prevent any automatic adding of a CAR reference.

3. When another reference is set, source localization will be unreliable/wrong

The user might not be aware of this, so a flag should be set if a non-CAR reference is applied. An error can then be generated during source localization.

Lets keep the code simple and make apply_reference always set this flag to True. This means that even if the user applies CAR using this function, it would be flagged as a non-CAR reference. Otherwise the function would have to do some trickery to determine whether the reference is CAR equivalent or require the user to specify this. Both methods are prone to error. Always setting the flag to True is another reason for removing the default behaviour of set_eeg_reference.

4. Computing the CAR on bad channels is bad

There should be a way to conveniently update the CAR projection after modifying the info['bads'] structure.

@dengemann @agramfort @t3on Do you agree on all points?

dengemann · 2014-11-28T08:10:24Z

regarding #1 the only thing is that you often do not know about bad channels before loading data, often you only tknow that at the epochs stage.

apart from that sounds good

On 28 Nov 2014, at 09:05, Marijn van Vliet notifications@github.com wrote:

Agreed.

Lets recap the EEG referencing arguments so far:

The default reference should be CAR
It should be straightforward for the user to apply another reference
When another reference is set, source localization will be unreliable/wrong
Computing the CAR on bad channels is bad

The default reference should be CAR

Upon loading a raw file, a CAR projector is automatically added for the EEG. Any bad channels already marked in the raw data are excluded.

It should be straightforward for the user to apply another reference

The apply_reference function applies a different reference, but not through an SSP projector (this is as far as I know only possible for CAR). The convenience functions set_eeg_reference and set_bipolar_reference wrap this function to provide a nice interface to the user.

When another reference is set, source localization will be unreliable/wrong

The user might not be aware of this, so a flag should be set if a non-CAR reference is applied. An error can then be generated during source localization.

Lets keep the code simple and make apply_reference always set this flag to True. This means that even if the user applies CAR using this function, it would be flagged as a non-CAR reference. Otherwise the function would have to do some trickery to determine whether the reference is CAR equivalent or require the user to specify this. Both methods are prone to error. Always setting the flag to True is another reason for removing the default behaviour of set_eeg_reference.

Lastly, if a CAR SSP projection is present in the data, it should be removed by this function. Otherwise things become very confusing for the user. When an instance of Epochs is created, the current behaviour is to add a CAR projection if not present. Instead, the flag should be checked and respected. The user should be able to use the set_eeg_reference to specify the data has already been referenced (some systems, like the Emotiv EPOC, do referencing at the driver level) and prevent any automatic adding of a CAR reference.

Computing the CAR on bad channels is bad

There should be a way to conveniently update the CAR projection after modifying the info['bads'] structure.

@dengemann @agramfort @t3on Do you agree on all points?

—
Reply to this email directly or view it on GitHub.

dengemann · 2014-11-28T08:11:28Z

yess, 4. is exactly it

On 28 Nov 2014, at 09:05, Marijn van Vliet notifications@github.com wrote:

Agreed.

Lets recap the EEG referencing arguments so far:

The default reference should be CAR
It should be straightforward for the user to apply another reference
When another reference is set, source localization will be unreliable/wrong
Computing the CAR on bad channels is bad

The default reference should be CAR

Upon loading a raw file, a CAR projector is automatically added for the EEG. Any bad channels already marked in the raw data are excluded.

It should be straightforward for the user to apply another reference

The apply_reference function applies a different reference, but not through an SSP projector (this is as far as I know only possible for CAR). The convenience functions set_eeg_reference and set_bipolar_reference wrap this function to provide a nice interface to the user.

When another reference is set, source localization will be unreliable/wrong

The user might not be aware of this, so a flag should be set if a non-CAR reference is applied. An error can then be generated during source localization.

Lets keep the code simple and make apply_reference always set this flag to True. This means that even if the user applies CAR using this function, it would be flagged as a non-CAR reference. Otherwise the function would have to do some trickery to determine whether the reference is CAR equivalent or require the user to specify this. Both methods are prone to error. Always setting the flag to True is another reason for removing the default behaviour of set_eeg_reference.

Lastly, if a CAR SSP projection is present in the data, it should be removed by this function. Otherwise things become very confusing for the user. When an instance of Epochs is created, the current behaviour is to add a CAR projection if not present. Instead, the flag should be checked and respected. The user should be able to use the set_eeg_reference to specify the data has already been referenced (some systems, like the Emotiv EPOC, do referencing at the driver level) and prevent any automatic adding of a CAR reference.

Computing the CAR on bad channels is bad

There should be a way to conveniently update the CAR projection after modifying the info['bads'] structure.

@dengemann @agramfort @t3on Do you agree on all points?

—
Reply to this email directly or view it on GitHub.

agramfort · 2014-11-28T10:27:10Z

The default reference should be CAR

Upon loading a raw file, a CAR projector is automatically added for the EEG. Any bad channels already marked in the raw data are excluded.

should we deprecate this default behavior?

It should be straightforward for the user to apply another reference

The apply_reference function applies a different reference, but not through an SSP projector (this is as far as I know only possible for CAR). The convenience functions set_eeg_reference and set_bipolar_reference wrap this function to provide a nice interface to the user.

I feel when CAR is asked the proj should be added and applied. It is
wrong to have 2 ways to do the same thing internally. But we can have
a common API to set the different references.

When another reference is set, source localization will be unreliable/wrong

The user might not be aware of this, so a flag should be set if a non-CAR reference is applied. An error can then be generated during source localization.

+1

Lets keep the code simple and make apply_reference always set this flag to True.

I would set it to True only if non CAR.

This means that even if the user applies CAR using this function, it would be flagged as a non-CAR reference.

I am -1 on this.

Lastly, if a CAR SSP projection is present in the data, it should be removed by this function. Otherwise things become very confusing for the user.

hide the complexity from the user.

does it seem doable?

Computing the CAR on bad channels is bad

There should be a way to conveniently update the CAR projection after modifying the info['bads'] structure.

+1. Any suggestion of how to do this?

wmvanvliet · 2014-11-28T11:32:21Z

should we deprecate this default behavior?

Probably not. It would open up the possibility for the user to forget to apply a reference. If we can make it easy for the user to set another reference, this default should not inconvenience the user.

I feel when CAR is asked the proj should be added and applied

But how should the user ask for CAR? Of course, the best method for the user would be to simply keep the default. But if the user really wants to explicitly ask for it, I was thinking of making him call the make_eeg_ref_proj function. Maybe it is a better idea to allow the user to specify CAR through the set_eeg_reference function as well. I will play with the API and get back to you.

hide the complexity from the user.
does it seem doable?

yes.

+1. Any suggestion of how to do this?

I was thinking about putting a system in place that checks the info dictionary. When it is changed (for example, channel names are updates, channels are added or removed) the dictionary is updated (nchans, ch_names, etc.) Such a system could also keep track of changes to the bad channels. Definitely not something for this PR, but something to think about.

wmvanvliet · 2014-11-28T11:44:01Z

what does the active flag of a projection actually do?

wmvanvliet · 2014-11-28T11:56:07Z

@agramfort how about this for an API to set_eeg_reference? Now it's backwards compatible, and used the correct method for applying CAR.

agramfort · 2014-11-28T12:39:41Z

what does the active flag of a projection actually do?

when a proj is active it means it has been applied to the data.

agramfort · 2014-12-12T16:40:44Z

besides LGTM

you'll need to update python_reference.rst + what's new

someone please have a look too.

larsoner · 2014-12-12T18:03:47Z

mne/io/meas_info.py

So this field always exists when reading info now...

yes, otherwise you'll have to check if 'custom_ref_applied' in info and info['custom_ref_applied'] every time you want to do something with the flag.

larsoner · 2014-12-12T18:09:03Z

Other than my minor concerns, LGTM.

agramfort · 2014-12-13T18:08:52Z

mne/io/base.py

pep257

One headline after """
then empty line and a full sentence / paragraph describing the function.

agramfort · 2014-12-13T18:20:07Z

also:

we should have an example that demonstrates the use of the function
you'll need to update what's new and python_reference.rst

that's it for me now.

wmvanvliet · 2014-12-16T14:10:45Z

Rebased.

agramfort · 2014-12-16T14:17:04Z

examples/plot_rereference_eeg.py

Use the mean of channels EEG 001 and EEG 002 as a reference

This adds a base function 'apply_reference' that can apply various different referencing schemes. Two convenience functions are also added that provide a convenient interface to the base function: - set_eeg_reference: apply a simple reference to all EEG channels - set_bipolar_reference: apply a bipolar reference (or more)

Raw and Epoch classes will no longer add an average reference if the `custom_ref_applied` flag was set. SSP projections will still attempt this, but a new assertion in `make_eeg_average_ref_proj` will raise an error if the flag was set. Before a CAR projection is added, the `info['custom_ref_applied']` flag is checked.

- Also check for duplicate channel names - Bilpolar channels can be named after the anode/cathode - Inform user of the chosen bipolar channel name

wmvanvliet · 2014-12-22T08:18:30Z

rebased (again), and tweaked the example a little to reduce memory consumption.

agramfort · 2014-12-23T13:51:29Z

closing in favor of #1713

@wmvanvliet see my last 4 commits.

dengemann reviewed Nov 25, 2014
View reviewed changes

mne/channels/channels.py Outdated

Copy link

Member

dengemann Nov 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wmvanvliet which problem does this solve?

agramfort reviewed Nov 26, 2014
View reviewed changes

mne/io/base.py Outdated

Copy link

Member

agramfort Nov 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eeg -> EEG

larsoner reviewed Dec 12, 2014
View reviewed changes

agramfort reviewed Dec 13, 2014
View reviewed changes

mne/io/base.py Outdated

Copy link

Member

agramfort Dec 13, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pep257

One headline after """
then empty line and a full sentence / paragraph describing the function.

wmvanvliet force-pushed the reference branch from fea301a to 268b44e Compare December 16, 2014 14:10

agramfort reviewed Dec 16, 2014
View reviewed changes

examples/plot_rereference_eeg.py Outdated

Copy link

Member

agramfort Dec 16, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the mean of channels EEG 001 and EEG 002 as a reference

wmvanvliet added 6 commits December 22, 2014 10:06

Add default channel name for bipolar references

ff0b1eb

- Also check for duplicate channel names - Bilpolar channels can be named after the anode/cathode - Inform user of the chosen bipolar channel name

Updating ref and what's new

c0a00a8

Some improvements to the docs

51ac222

Make apply_reference a private function

21c2227

wmvanvliet force-pushed the reference branch from 7b6f1e6 to 21c2227 Compare December 22, 2014 08:10

Reduced memory consumption of rerefencing example

cad7e35

agramfort closed this Dec 23, 2014

wmvanvliet deleted the reference branch January 5, 2015 09:31

Uh oh!

Add more referencing options #1670

Add more referencing options #1670

Uh oh!

Conversation

wmvanvliet commented Nov 25, 2014

Uh oh!

dengemann Nov 25, 2014

Choose a reason for hiding this comment

Uh oh!

coveralls commented Nov 26, 2014

Uh oh!

agramfort Nov 26, 2014

Choose a reason for hiding this comment

Uh oh!

agramfort commented Nov 26, 2014

Uh oh!

wmvanvliet commented Nov 26, 2014

Uh oh!

agramfort commented Nov 26, 2014

Uh oh!

dengemann commented Nov 26, 2014

Uh oh!

wmvanvliet commented Nov 26, 2014

Uh oh!

agramfort commented Nov 26, 2014

Uh oh!

dengemann commented Nov 26, 2014

Uh oh!

wmvanvliet commented Nov 26, 2014

Uh oh!

agramfort commented Nov 26, 2014

Uh oh!

wmvanvliet commented Nov 26, 2014

Uh oh!

agramfort commented Nov 26, 2014

Uh oh!

wmvanvliet commented Nov 28, 2014

1. The default reference should be CAR

2. It should be straightforward for the user to apply another reference

3. When another reference is set, source localization will be unreliable/wrong

4. Computing the CAR on bad channels is bad

Uh oh!

dengemann commented Nov 28, 2014

Uh oh!

dengemann commented Nov 28, 2014

Uh oh!

agramfort commented Nov 28, 2014

Uh oh!

wmvanvliet commented Nov 28, 2014

Uh oh!

wmvanvliet commented Nov 28, 2014

Uh oh!

wmvanvliet commented Nov 28, 2014

Uh oh!

agramfort commented Nov 28, 2014

Uh oh!

agramfort commented Dec 12, 2014

Uh oh!

larsoner Dec 12, 2014

Choose a reason for hiding this comment

Uh oh!

wmvanvliet Dec 13, 2014

Choose a reason for hiding this comment

Uh oh!

larsoner commented Dec 12, 2014

Uh oh!

agramfort Dec 13, 2014

Choose a reason for hiding this comment

Uh oh!

agramfort commented Dec 13, 2014

Uh oh!

wmvanvliet commented Dec 16, 2014

Uh oh!

agramfort Dec 16, 2014

Choose a reason for hiding this comment

Uh oh!

wmvanvliet commented Dec 22, 2014

Uh oh!

agramfort commented Dec 23, 2014