ENH: Allow automated compilation of metadata

[hoechenberger]

This PR adds mne.epochs.make_metadata(), which allows for the automated construction of metadata for use in Epochs.

The users passes an events array, an event name -> event ID mapping, a time period and a sampling frequency. The function will then

• iterate over all events specified in the mapping,
• determine which events fall into the specified time period (relative to the current – time-locked – event), and
• will add them to the metadata in two cells: one contains a boolean value indicating whether the event in question is present in the respective time period / Epoch, and one with a float value of the event time in seconds, relative to the time-locked event.

It will then return the constructed metadata and a matching subset of the events array. These values can then be readily used to construct Epochs of arbitrary durations, while ensuring all of them will be assigned the desired metadata (which may or may not contain events from a different time range around the time-locked event)

Example:

import os
import mne

sample_data_folder = mne.datasets.sample.data_path()
sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                    'sample_audvis_raw.fif')
raw = mne.io.read_raw_fif(sample_data_raw_file, verbose=False).crop(tmax=120)
tmax = 1
tmin = -0.5
sfreq = raw.info['sfreq']

all_events = mne.find_events(raw, stim_channel='STI 014')
event_id = {'auditory/left': 1, 'auditory/right': 2, 'visual/left': 3,
            'visual/right': 4, 'face': 5, 'button': 32}

metadata, events = mne.epochs.make_metadata(
    events=all_events, event_id=event_id, tmin=tmin, tmax=tmax, sfreq=sfreq)

epochs = mne.Epochs(raw, events, tmin=tmin, tmax=tmax, event_id=event_id,
                    preload=True, metadata=metadata)
epochs.metadata

Conditionally selecting a subset of Epochs is easy thanks to Pandas query strings:

epochs['`visual/right` and button']

<Epochs |  7 events (all good), -0.499488 - 1.00064 sec, baseline [-0.499488, 0] sec, ~21.4 MB, data loaded, with metadata,
 'auditory/left': 2
 'button': 3
 'face': 1
 'visual/right': 1>

epochs['`visual/right` and not `visual/left`']

<Epochs |  80 events (all good), -0.499488 - 1.00064 sec, baseline [-0.499488, 0] sec, ~210.3 MB, data loaded, with metadata,
 'auditory/left': 38
 'button': 3
 'face': 1
 'visual/right': 38>

This sort-of addresses #8360 as well.

I'd like to hear your opinions on this.

I've specifically tried to create a function that produces very generic data to ensure maximum versatility. I believe this set of metadata can also serve as a skeleton / starting point for users, as they're free to manipulate the data further before adding it to their Epochs.

cc @SophieHerbst

[hoechenberger]

WIP: Handling repeated events within the same time period

[agramfort]

@hoechenberger can you make / update an example that gets rendered in the doc so people can get a quick way to get started?

I am personally super excited about this. It makes certain things suddenly so much easier to do !

[hoechenberger]

I will add it to a tutorial @agramfort

[drammock]

I am personally super excited about this. It makes certain things suddenly so much easier to do !

Agree, this looks like a very useful addition

[larsoner]

Looks like a useful addition to me. Probably worth adding to an example or two to see how it's useful. Maybe an epochs tutorial and/or metadata tutorial, then also maybe the reaction times example if it actually helps there?

[agramfort]

besides some documentation I think it's good to go. I've already seen a successful use of this code for data with reaction times.

[hoechenberger]

@larsoner @drammock @agramfort
This is ready for review.

I suggest you start with the tutorial I created to get an idea of how the various parameters work:
https://25350-1301584-gh.circle-artifacts.com/0/dev/auto_tutorials/epochs/plot_40_autogenerate_metadata.html

The tutorial can probably be made more concise and its language can certainly be improved. Any suggestions welcome!

[drammock]

I've reviewed only the docstring so far.

[hoechenberger]

Thank you for your review, @drammock! I will try to address your comments very soon.

[hoechenberger]

Sorry I have to bug you all again, but when implementing the discussed changes and playing through the tutorial several times, I now feel the urge to add a slight modification.

In the beginning, we had two columns per stimulus:

•  one named stimulus, containing booleans indicating whether the stimulus was present in the current time window;
• and the other named stimulus_time with the latency.

We then decided to drop the boolean columns due to their redundancy, as absence can be indicated by NaN in the latency columns too. All good so far.

But we also opted to rename to drop the _time suffix from the latency columns' names. So what used to be stimulus_time is now simply stimulus. And this is what bothers me now.
I now have code looking like this:

epochs['response < 1.5 & first_response == "left"']

And I think this is semantically less … pleasant than:

epochs['response_time < 1.5 & first_response == "left"']

So here is my simple question: Can we have the _time back (or, if you will, _latency or whichever suffix might seem appropriate)? We could even make the suffix user-configurable if we cannot reach a clear consensus here. But I'd love to have a way to have my columns containing times named accordingly.

Looking forward to hearing your thoughts!

😇🦄🥸

[drammock]

I think this is semantically less … pleasant

it doesn't bother me as much as it seems to bother you. For me, downsides that outweigh it are:

1. not being able to see my whole dataframe in the terminal because the column names are all 5 characters longer than I think they need to be.
2. having to type the extra underscore & word in a context where tab completion is unavailable (i.e., within a query string)

If users really want a suffix there is pandas.DataFrame.add_suffix. If we really can't reach agreement I would settle for a suffix='_time' parameter so I can specify suffix=''.

[agramfort]

I have no opinion :)

…

[cbrnr]

I agree with @drammock. It's pretty clear without _time and the benefits are more important to me.

[agramfort]

let's remove the _time then and l merge during the day :)

…

[hoechenberger]

Whoops, day is almost over, sorry ;) I'll finish it up within the next hours. Thanks for your feedback!

[hoechenberger]

Doc artifacts:

Tutorial:
https://25600-1301584-gh.circle-artifacts.com/0/dev/auto_tutorials/epochs/plot_40_autogenerate_metadata.html#sphx-glr-auto-tutorials-epochs-plot-40-autogenerate-metadata-py

API docs:
https://25600-1301584-gh.circle-artifacts.com/0/dev/generated/mne.epochs.make_metadata.html

[agramfort]

thx @hoechenberger

[drammock]

Many thanks for this! I think it will be really useful for a lot of people and I appreciate your patience with all the API back-and-forth.