ENH: stats for ERF decoding and time generalization

[dengemann]

With @kingjr we're currently thinking about the right way to handle cross-trial significance testing for the analyses demonstrated in our decoding examples. This is basically overlapping with one of our GSOC topics, and the discussion in the comments over here. Now practically, permutation testing or bootstrapping seems pretty heavy, if it means that you resample every model you fit. This will render these kind of analyses intractable. On the flip side there seems to be a problem in using any test that makes assumptions about the degrees of freedom, as for single trial probability output, for example, the samples are not independend because of cross-validation. It seems, instead of bootstrapping the entire decoding or incorporating permutation testing into cross-validation, as suggested by Martin Hebart (first comment here), we could just compute a permutation test post-hoc on the predictions. This would be much more tractable.
I was wondering what you think might be the right way to go. Also for the GSOC project it would be nice to have some discussion about this.

cc @agramfort @Eric89GXL @banilo

[dengemann]

cc @deep-introspection

[dengemann]

cc @flrgsr

[deep-introspection]

@dengemann I like Hebart's solution but this may be computationally demanding... By the way, to what extend MNE is handling distributed computing?

[dengemann]

By the way, to what extend MNE is handling distributed computing?

DYI style ...

[agramfort]

with joblib

[dengemann]

with joblib

@agramfort I think @deep-introspection is referring to distributed computing across machines, MPI style

[deep-introspection]

@dengemann Indeed, but thanks @agramfort for the joblib tip, sounds pretty cool!

[agramfort]

see clusterlib it's WIP though

[dengemann]

some more food for thoughts:

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=21D16E681FF16B0BA91D741A63806A31?doi=10.1.1.29.5194&rep=rep1&type=pdf

http://epub.ub.uni-muenchen.de/4134/1/tr030.pdf

[dengemann]

Salzberg says:

"(It may be tempting, because it is easy to do with powerful computers, to run many cross
validations on the same data set, and report each cross validation as a single trial. However this would not produce valid statistics, because the trials in such a design are highly
interdependent.)."

On the otherhand this is essentially what Eugster and colleagues propose and what is implemented in the caret resamples function If I'm not mistaken. It essentially computes confidence intervals across resamples based on a t-statistic. Here nuances refer to using bootstrap for model comparison instead of e.g. a repaeated 10-fold CV.

(cc @topepo)

[dengemann]

More materials:

http://stats.stackexchange.com/questions/17408/how-to-assess-statistical-significance-of-the-accuracy-of-a-classifier

[dengemann]

cc @cmoutard

[choldgraf]

Seeing if you guys thought up anything on this issue. I'm trying to gauge whether my decoding scores are "significant" or not as has been mentioned here, doing permutation testing on something like this would take a long time :)

[topepo]

Without knowing too much about the details of the problem, my thoughts are:

• comparisons between resamples is probably preferable since it can drastically improve the power of the comparison (relative to a single static test set). Controlling for the resample-to-resample variability, which can be huge, is easy to do.
• permutation tests, depending on how yo do them, tend to test the hypothesis that the signal is above the noise. If you are using them to evaluate a single model, knowing that the performance is better than noise can be a very low bar and measuring the uncertainty of the performance metric (via confidence or credible intervals) is probably a better idea. A permutation test might be more applicable if you jointly permute two models and see if the difference in performance between them is above the experimental noise.