artifact detection #7096

AdoNunes · 2019-11-21T16:35:47Z

I have added (temporally in the wrong place) a function script to mark suspicious channels of being bad, a muscle artifact detection (in practice almost the same as if Fieltrip) and movement annotation and head position trans correction.

The muscle and movement might be crucial in some datasets when Maxfilter is not applied and currently mne-python does not provide these functions.

codecov · 2019-11-21T17:58:57Z

Codecov Report

❗ No coverage uploaded for pull request base (master@23dd105). Click here to learn what that means.
The diff coverage is 93.85%.

@@            Coverage Diff            @@
##             master    #7096   +/-   ##
=========================================
  Coverage          ?   89.79%           
=========================================
  Files             ?      447           
  Lines             ?    80506           
  Branches          ?    12873           
=========================================
  Hits              ?    72289           
  Misses            ?     5390           
  Partials          ?     2827

agramfort · 2019-11-21T19:37:23Z

@AdoNunes can you your file plot_ and not demo_ so it gets executed thanks

agramfort · 2019-11-21T20:17:33Z

@AdoNunes it needs to be in examples/preprocessing folder so it’s rendered by circle ci

…python into artifact_detection merge origin

AdoNunes · 2019-11-21T21:14:05Z

@AdoNunes it needs to be in examples/preprocessing folder so it’s rendered by circle ci

sorry, I got it now....

AdoNunes · 2019-11-22T02:26:43Z

Why I am getting the ci errors?
Like this:
generating gallery for auto_examples/preprocessing... [ 5%] plot_artifact_detection.py
Killed
Makefile:49: recipe for target 'html_dev-pattern' failed
make: *** [html_dev-pattern] Error 137
Exited with code 2

jasmainak · 2019-11-22T02:43:00Z

I suspect it's a memory issue. Can you check using a memory profiler how much memory you consume?

larsoner · 2019-11-22T05:12:57Z

To follow what @jasmainak said, you should try to keep examples below 1.5 GB memory consumption, otherwise you risk CircleCI failing. mprof run my_example.py then mprof plot will show you consumption. If you can get away with using 30 sec or 60 sec of data, do read_raw_ctf(fname).crop(None, 30).load_data() to save memory. It's why a lot of our examples start that way.

AdoNunes · 2019-11-22T14:50:18Z

That it's tight! It peaks at 1.7Gb but then it goes down.

…python into artifact_detection conflict Conflicts: examples/preprocessing/plot_artifact_detection.py

jasmainak · 2019-11-22T15:59:53Z

Try to see if you're making any redundant copies and get rid of them. You can also use preload=False to save memory.

AdoNunes · 2019-11-22T16:43:31Z

Try to see if you're making any redundant copies and get rid of them. You can also use preload=False to save memory.

By shortening even more, changing the order and decimating first then notching it worked :)

I will fix the ::doc things and I will resubmit

AdoNunes · 2019-11-22T19:53:23Z

I am getting an error and I don't know who is wrong.

When I crop de raw, say from (100, 300). The raw.times starts at 0, although, the raw.first_time is 100. Is that how it is supposed to be?

Because, then the function _calculate_head_pos_ctf returns "quats", the quats[:, 0] is the time, and in chpi.py line 264 -> quats[:, 0] += raw._first_time. So the quats time has an offset that does not match the cropped raw.times and this gives me wrong annotation timings.

Any insights?

jasmainak · 2019-11-23T16:56:50Z

I don't know about quats but it looks like you just need to handle it correctly by adding raw.first_time in the right places. You might find this tutorial useful: https://mne.tools/stable/auto_tutorials/raw/plot_30_annotate_raw.html

Let's make it work so that CircleCI renders something with plots and then we can worry about these details later.

examples/preprocessing/plot_artifact_detection.py

jasmainak · 2019-11-23T16:58:38Z

examples/preprocessing/plot_artifact_detection.py

+raw.notch_filter([60, 120, 180, 240]).resample(300, npad="auto")
+
+# Detect bad channels
+bad_chns = detect_bad_channels(raw, zscore_v=4, method='both', t1=0, t2=190,


bad_chns -> bad_chs to be consistent with rest of repo

jasmainak · 2019-11-23T16:58:58Z

mne/artifact_detection.py

+        decreased local correlation with other channels"""
+
+    # set recording length
+    Fs = raw.info['sfreq']


Fs -> sfreq

jasmainak · 2019-11-23T16:59:58Z

mne/artifact_detection.py

+    # Get data
+    raw_copy = raw.copy().crop(t1x, t2x).load_data()
+    raw_copy = raw_copy.pick_types(meg=True, ref_meg=False)\
+        .filter(1, 45).resample(150, npad='auto')


You want to hardcode these numbers?

jasmainak · 2019-11-23T17:00:14Z

mne/artifact_detection.py

+    data_chans = raw_copy.get_data()
+
+    # Get channel distances matrix
+    chns_locs = np.asarray([x['loc'][:3] for x in


chns_locs -> ch_locs

jasmainak · 2019-11-23T17:01:41Z

mne/artifact_detection.py

+def detect_bad_channels(raw, zscore_v=4, method='both', t1=30, t2=220,
+                        neigh_max_distance=.035):
+    """ detect bad channels based on z-score amplitude deviation or/ and
+        decreased local correlation with other channels"""


Add a docstring to explain what each of these parameters are

jasmainak · 2019-11-23T17:02:19Z

mne/artifact_detection.py

+"""
+Created on Tue Nov 19 09:40:37 2019
+
+@author: an512


This is not kosher. Put this file under mne/preprocessing/ and then format the header like the other files.

I thought that these scripts were temporary. Fixing it!

jasmainak · 2019-11-23T17:02:36Z

mne/artifact_detection.py

+"""
+
+from scipy.stats import zscore
+import mne


mne import should come after numpy and scipy

Actually you shouldn't import mne at all unless you're writing a tutorial or a test. You should do relative imports.

larsoner · 2020-01-24T17:05:37Z

@mmagnuski let me know if you're happy now and I'll have a look

mmagnuski · 2020-01-24T18:22:49Z

@AdoNunes weird error, I don't have a clue, but @larsoner probably knows. :)

mmagnuski · 2020-01-24T18:25:36Z

oh, I had an old unrefreshed tab it seems :) I don't have anything to add here, but don't have experience with MEG, so I mostly commented on code style and docs.

mmagnuski · 2020-01-24T18:29:01Z

Do we need a test here is the example enough?

larsoner · 2020-01-24T18:55:37Z

Yes we need some tests. @AdoNunes you can make use of this existing pos_fname and associated raw file in the testing data:

https://github.com/mne-tools/mne-python/blob/master/mne/preprocessing/tests/test_maxwell.py#L65

Basically choose some thresholds that should mark segments as bad and verify that they are indeed marked as bad. You can iteratively test pos/velocity translational/rotational since it should have all of these types in a short window (we asked the subject to move their head for this recording)

larsoner · 2020-01-27T14:52:28Z

mne/preprocessing/tests/test_artifact_detection.py

+                          [-0.01964518, 0.13086313, 0.99120578, 0.07258289],
+                          [0., 0., 0., 1.]])
+
+assert_allclose(dev_head_t_ori, dev_head_t['trans'], rtol=1e-5, atol=0)


All of this needs to be nested inside a function like def test_artifact_detection(): otherwise it will be run on test collection (see how we do it in other files).

larsoner · 2020-01-27T14:53:27Z

mne/preprocessing/tests/test_artifact_detection.py

+
+with warnings.catch_warnings(record=True):
+    warnings.simplefilter("ignore")
+    raw = read_raw_fif(raw_fname, allow_maxshield=True).load_data()


allow_maxshield='yes' then you don't need any warnings gymnastics (see existing cHPI code for example)

mne/tests/test_import_nesting.py

mne/preprocessing/tests/test_artifact_detection.py

examples/preprocessing/plot_artifact_detection.py

AdoNunes · 2020-01-28T18:38:54Z

Sure!

…

On Tue., Jan. 28, 2020, 1:31 p.m. Eric Larson ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In examples/preprocessing/plot_artifact_detection.py <#7096 (comment)>: > +plt.axhline(y=mean_distance_limit, color='r') +plt.xlabel('time s') +plt.ylabel('distance m') +plt.title('cHPI w.r.t mean recording head position ') +plt.show(block=False) + +############################################################################### +# Plot raw data with annotated movement +# ------------------------------------------------------------------ +raw.set_annotations(annotation_movement) +raw.plot(n_channels=100, duration=20) + + +# After checking the annotated movement artifacts, calculate the new transform +new_dev_head_t = compute_average_dev_head_t(raw, pos) +raw.info['dev_head_t'] = new_dev_head_t @AdoNunes <https://github.com/AdoNunes> I have some ideas for improving this script, okay if I push to your branch? It would be faster than writing a bunch of comments — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7096?email_source=notifications&email_token=AEILW3W4JG6VAGFHCU52WD3RAB2XFA5CNFSM4JQFA22KYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCTLIY7Y#pullrequestreview-349604991>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEILW3WZFCWPIPGKDYU3T43RAB2XFANCNFSM4JQFA22A> .

I thought it would be didactically illustrative to show the original head position compared to the averaged. Let's see what Travis complains about...

agramfort

this is rendered page https://17718-1301584-gh.circle-artifacts.com/0/dev/auto_examples/preprocessing/plot_artifact_detection.html#sphx-glr-auto-examples-preprocessing-plot-artifact-detection-py

LGTM

AdoNunes · 2020-01-29T14:34:50Z

Great!!!!
It took longer than expected! I hope people will use these functions in the future!
Thanks for the help

agramfort · 2020-01-29T15:07:42Z

it will be much faster for your next one I am sure :)

…

larsoner · 2020-01-29T15:15:08Z

Thanks @AdoNunes !

* artifact detection * art def * Rename file * style corrections * cropped * example cropped * flake corrections * added docstrings * fixed doc style errors * updatex example * fix docstring * added offset removal * added compute_average_dev_head_t * New head pos trans and annotate movement * Copy raw instance, corrected pos time * docstring fix * movement artifact only * Updated func names and example * enh doc & quat average * define norm * docstring correction * revive CI * quat 3 * docstrings improvement * logger and docstrings * docstrings * docstring, reference, latest * funct import * add func in init * import typo * quoted BAD_ * docstrings * FIX: Nesting test * added test * read raw * test def * docstring * docstring * test function * WIP: Tweaks * orig head pos line I thought it would be didactically illustrative to show the original head position compared to the averaged. Let's see what Travis complains about... * docstrings * warning to logger Co-authored-by: Eric Larson <larson.eric.d@gmail.com>

artifact detection

3feef62

AdoNunes added 2 commits November 21, 2019 14:26

Merge remote-tracking branch 'upstream/master' into artifact_detection

676eaa5

art def

86e5706

Rename file

5403cac

AdoNunes added 2 commits November 21, 2019 15:29

Merge branch 'artifact_detection' of https://github.com/AdoNunes/mne-…

c100cdc

…python into artifact_detection merge origin

style corrections

4bee23c

cropped

7c67844

AdoNunes added 3 commits November 22, 2019 09:51

example cropped

ef9957b

Merge branch 'artifact_detection' of https://github.com/AdoNunes/mne-…

2fb6010

…python into artifact_detection conflict Conflicts: examples/preprocessing/plot_artifact_detection.py

flake corrections

4226e67

jasmainak reviewed Nov 23, 2019

View reviewed changes

examples/preprocessing/plot_artifact_detection.py Outdated Show resolved Hide resolved

jasmainak reviewed Nov 23, 2019

View reviewed changes

AdoNunes added 2 commits January 26, 2020 21:06

added test

492f8e9

read raw

0084d5e

larsoner reviewed Jan 27, 2020

View reviewed changes

AdoNunes added 3 commits January 27, 2020 19:18

test def

652a5a9

docstring

ed6d369

docstring

9aa1c52

agramfort reviewed Jan 28, 2020

View reviewed changes

mne/tests/test_import_nesting.py Show resolved Hide resolved

agramfort reviewed Jan 28, 2020

View reviewed changes

mne/preprocessing/tests/test_artifact_detection.py Outdated Show resolved Hide resolved

larsoner reviewed Jan 28, 2020

View reviewed changes

mne/preprocessing/tests/test_artifact_detection.py Outdated Show resolved Hide resolved

mne/preprocessing/tests/test_artifact_detection.py Outdated Show resolved Hide resolved

AdoNunes added 2 commits January 28, 2020 11:59

test function

23dd105

Merge branch 'master' into artifact_detection

94f2158

larsoner reviewed Jan 28, 2020

View reviewed changes

examples/preprocessing/plot_artifact_detection.py Show resolved Hide resolved

larsoner and others added 4 commits January 28, 2020 15:40

WIP: Tweaks

333adb0

orig head pos line

8eee9c5

I thought it would be didactically illustrative to show the original head position compared to the averaged. Let's see what Travis complains about...

docstrings

4a0e847

warning to logger

e20b867

agramfort approved these changes Jan 29, 2020

View reviewed changes

larsoner merged commit cce705b into mne-tools:master Jan 29, 2020

AdoNunes mentioned this pull request Feb 28, 2020

ENH: what to do with muscle artifacts #7359

Closed

Uh oh!

artifact detection #7096

artifact detection #7096

Uh oh!

Conversation

AdoNunes commented Nov 21, 2019

Uh oh!

codecov bot commented Nov 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agramfort commented Nov 21, 2019

Uh oh!

agramfort commented Nov 21, 2019

Uh oh!

AdoNunes commented Nov 21, 2019

Uh oh!

AdoNunes commented Nov 22, 2019

Uh oh!

jasmainak commented Nov 22, 2019

Uh oh!

larsoner commented Nov 22, 2019

Uh oh!

AdoNunes commented Nov 22, 2019

Uh oh!

jasmainak commented Nov 22, 2019

Uh oh!

AdoNunes commented Nov 22, 2019

Uh oh!

AdoNunes commented Nov 22, 2019

Uh oh!

jasmainak commented Nov 23, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner commented Jan 24, 2020

Uh oh!

mmagnuski commented Jan 24, 2020

Uh oh!

mmagnuski commented Jan 24, 2020

Uh oh!

mmagnuski commented Jan 24, 2020

Uh oh!

larsoner commented Jan 24, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdoNunes commented Jan 28, 2020 via email

Uh oh!

agramfort left a comment

Choose a reason for hiding this comment

Uh oh!

AdoNunes commented Jan 29, 2020

Uh oh!

agramfort commented Jan 29, 2020 via email

Uh oh!

codecov bot commented Nov 21, 2019 •

edited

Loading