WIP: Add temporal derivative distribution repair algorithm #7556

rob-luke · 2020-04-03T07:49:04Z

Reference issue

As discussed in #7057

What does this implement/fix?

Adds temporal derivative distribution repair (TDDR) technique.
Adds new example showing how to apply nirs artefact correction (with plan to expand as new methods are implemented).

Additional information

I have taken the code directly from the author with permission.

I have kept the code as close to the original from the author as possible, and then used a wrapper function. I did this as the code was very nice. Also if the upstream version changes it is then easier to move the changes here.

mne/preprocessing/nirs/_tddr.py

codecov · 2020-04-03T09:03:16Z

Codecov Report

Merging #7556 into master will decrease coverage by 0.01%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##           master    #7556      +/-   ##
==========================================
- Coverage   90.13%   90.12%   -0.02%     
==========================================
  Files         450      452       +2     
  Lines       82728    82800      +72     
  Branches    13076    13083       +7     
==========================================
+ Hits        74565    74621      +56     
- Misses       5342     5351       +9     
- Partials     2821     2828       +7

rob-luke · 2020-04-03T09:04:07Z

Thanks for the early feedback @agramfort.

doc/references.bib

examples/preprocessing/plot_fnirs_artifact_removal.py

mne/preprocessing/nirs/__init__.py

mne/preprocessing/nirs/temporal_derivative_distribution_repair.py

mne/preprocessing/tests/test_temporal_derivative_distribution_repair.py

rob-luke · 2020-04-04T06:32:13Z

@larsoner could you please review again. I think I made all your suggested changes and the CI segmentation error looks unrelated.

larsoner

Just a few more minor comments

Example looks good. Would it be worth adding to tutorials/plot_70_fnirs_processing.py in addition to the example? If we're thinking of that as the gold standard for people to follow it seems like it, even if you just add a cross-link to the other example and the :func:.

larsoner · 2020-04-04T14:03:58Z

mne/preprocessing/nirs/_temporal_derivative_distribution_repair.py

+    if not len(pick_types(raw.info, fnirs='fnirs_od')):
+        raise RuntimeError('TDDR should be run on optical density data.')
+
+    raw._data = _TDDR(raw._data.T, raw.info['sfreq']).T


This will operate on all of the data even if there are non-fNIRS channels. So probably needs picks incorporated

Thanks. Good catch.

larsoner · 2020-04-04T14:05:18Z

mne/preprocessing/nirs/_temporal_derivative_distribution_repair.py

+    if len(signal.shape) != 1:
+        for ch in range(signal.shape[1]):
+            signal[:, ch] = _TDDR(signal[:, ch], sample_rate)
+        return signal


If this is how it operates, it would be simpler not to make this modification, and do the looping over picks directly in temporal_derivative_distribution_repair. It will hopefully keep this function just a copy-paste drop-in (?) and also takes care of the picks problem at the same time.

This still required the name change to keep flake happy unfortunately.

larsoner · 2020-04-04T14:06:39Z

... also needs a rebase

mne/preprocessing/nirs/_temporal_derivative_distribution_repair.py

rob-luke · 2020-04-05T03:11:37Z

@frankfishburn this code is ready to be merged in to MNE. You mentioned you probably wouldn't have time to take a look, so this is just an FYI.

agramfort

besides LGTM

https://19168-1301584-gh.circle-artifacts.com/0/dev/auto_examples/preprocessing/plot_fnirs_artifact_removal.html#sphx-glr-auto-examples-preprocessing-plot-fnirs-artifact-removal-py

examples/preprocessing/plot_fnirs_artifact_removal.py

rob-luke · 2020-04-05T09:51:22Z

Thanks @agramfort

larsoner · 2020-04-05T12:56:52Z

Thanks @rob-luke

* upstream/master: (1522 commits) FIX: Show bug MRG, FIX: Datetime call in gdf 2.x age calculation (mne-tools#7581) DOC: Simplify Darwin installation (mne-tools#7584) MRG, ENH: Allow picking without preload (mne-tools#7507) DOC: Document anonymization better (mne-tools#7587) Rework _Brain show (mne-tools#7580) DOC: Fixes in tutorial (mne-tools#7579) ENH: muscle artifact detection (mne-tools#7407) MRG: Remove toolbars in PyVista plotter (mne-tools#7572) WIP: Deregister plotter from the figure list in close() (mne-tools#7573) MRG: Fix mouse wheel event in _TimeViewer (mne-tools#7563) FIX: Fix toggle all (mne-tools#7567) MRG, FIX: parallel n_jobs check (mne-tools#7566) Rename artifact detection to movement detection (mne-tools#7569) ENH: Update spelling check [ci skip] (mne-tools#7565) MRG, ENH: Dont require preload for raw resample (mne-tools#7508) MRG: Add interpolation for NIRS signals (mne-tools#7428) WIP: Add temporal derivative distribution repair algorithm (mne-tools#7556) DOC: fix link in docstr [skip ci] (mne-tools#7562) ENH: Custom figure title when plotting Dipole locations (mne-tools#7558) ...

agramfort reviewed Apr 3, 2020

View reviewed changes

mne/preprocessing/nirs/_tddr.py Outdated Show resolved Hide resolved

mne/preprocessing/nirs/_tddr.py Show resolved Hide resolved

drammock reviewed Apr 3, 2020

View reviewed changes

doc/references.bib Outdated Show resolved Hide resolved

doc/references.bib Outdated Show resolved Hide resolved

larsoner reviewed Apr 3, 2020

View reviewed changes

rob-luke force-pushed the fnirs_tddr branch 3 times, most recently from 6fb24ee to 04337a1 Compare April 4, 2020 03:32

rob-luke changed the title ~~WIP: Add temporal derivative distribution repair algorithm~~ MRG: Add temporal derivative distribution repair algorithm Apr 4, 2020

larsoner reviewed Apr 4, 2020

View reviewed changes

mne/preprocessing/nirs/_temporal_derivative_distribution_repair.py Outdated Show resolved Hide resolved

rob-luke changed the title ~~MRG: Add temporal derivative distribution repair algorithm~~ WIP: Add temporal derivative distribution repair algorithm Apr 4, 2020

rob-luke force-pushed the fnirs_tddr branch from 7fb1787 to ba60be7 Compare April 5, 2020 01:37

Add temporal derivative distribution repair algorithm

0c67858

rob-luke force-pushed the fnirs_tddr branch from ba60be7 to 0c67858 Compare April 5, 2020 02:20

Fix grammar in fnirs tutorial

8f015a9

agramfort approved these changes Apr 5, 2020

View reviewed changes

examples/preprocessing/plot_fnirs_artifact_removal.py Outdated Show resolved Hide resolved

Fix typo in fnirs tutorial

f6ebc65

larsoner merged commit 50f9f7b into mne-tools:master Apr 5, 2020

rob-luke mentioned this pull request May 19, 2020

Rename temporal_derivative_distribution_repair to tddr #7798

Closed

rob-luke mentioned this pull request Sep 9, 2020

Porting of TDDR to MNE frankfishburn/TDDR#1

Closed

HanBnrd mentioned this pull request Dec 13, 2021

ENH: TDDR for Hb type fNIRS data #10125

Merged

Uh oh!

WIP: Add temporal derivative distribution repair algorithm #7556

WIP: Add temporal derivative distribution repair algorithm #7556

Uh oh!

Conversation

rob-luke commented Apr 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference issue

What does this implement/fix?

Additional information

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Apr 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rob-luke commented Apr 3, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rob-luke commented Apr 4, 2020

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

larsoner Apr 4, 2020

Choose a reason for hiding this comment

Uh oh!

rob-luke Apr 4, 2020

Choose a reason for hiding this comment

Uh oh!

larsoner Apr 4, 2020

Choose a reason for hiding this comment

Uh oh!

rob-luke Apr 4, 2020

Choose a reason for hiding this comment

Uh oh!

larsoner commented Apr 4, 2020

Uh oh!

Uh oh!

rob-luke commented Apr 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agramfort left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rob-luke commented Apr 5, 2020

Uh oh!

larsoner commented Apr 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rob-luke commented Apr 3, 2020 •

edited

Loading

codecov bot commented Apr 3, 2020 •

edited

Loading

rob-luke commented Apr 5, 2020 •

edited

Loading