MRG: Add Brain class based on _Renderer

[GuillaumeFavelier]

EDIT: In the last commit, Brain has been renamed into _Brain from mne.viz._brain. The code snippets below are not up to date.

While work is still in progress in #6232 to fix transparency issues, I thought it would be great to move forward the refactoring of the source code based on PySurfer. This PR introduces the class Brain using the 3d abstraction layer _Renderer.

This PR is just the first step, the goal after is to refactor/test and replace parts of the source code relying on PySurfer.

Now I show some visualizations and code snippets using the latest 3d backend merged recently PyVista:

Basic Visualization

import mne
import os.path as path
from mne.viz import Brain, use_3d_backend
data_path = mne.datasets.sample.data_path()

subject_id = 'sample'
subjects_dir = path.join(data_path, 'subjects')
hemi = 'both'
surf = 'inflated'

with use_3d_backend("pyvista"):
    brain = Brain(subject_id, hemi, surf, size=300, subjects_dir=subjects_dir)
    brain.show()

Activation Data Visualization

import mne
import os.path as path
from mne.viz import Brain, set_3d_backend
data_path = mne.datasets.sample.data_path()

subject_id = 'sample'
subjects_dir = path.join(data_path, 'subjects')
surf = 'inflated'
act_data = path.join(data_path, 'MEG/sample/sample_audvis-meg-eeg')
stc = mne.read_source_estimate(act_data)
hemi = 'lh'
hemi_data = stc.data[:len(stc.vertices[0]), 10]
hemi_vertices = stc.vertices[0]
fmin = stc.data.min()
fmax = stc.data.max()

set_3d_backend("pyvista")
brain = Brain(subject_id, hemi, surf, size=300, subjects_dir=subjects_dir)
brain.add_data(hemi_data, fmin=fmin, hemi=hemi, fmax=fmax, colormap='hot',
               vertices=hemi_vertices, colorbar=False)

brain.show()

Display ROI Values

import numpy as np
import os.path as path
import mne
from mne.viz import Brain, set_3d_backend
import os
import nibabel as nib

subject_id = "fsaverage"
data_path = mne.datasets.sample.data_path()
subjects_dir = path.join(data_path, 'subjects')
hemi = "lh"
surf = "inflated"

set_3d_backend("pyvista")
brain = Brain(subject_id, hemi, surf, background=(0, 0, 0),
              subjects_dir=subjects_dir)

aparc_file = os.path.join(subjects_dir,
                          subject_id, "label",
                          hemi + ".aparc.a2009s.annot")
labels, ctab, names = nib.freesurfer.read_annot(aparc_file)
rs = np.random.RandomState(4)
roi_data = rs.uniform(.5, .8, size=len(names))
vtx_data = roi_data[labels]
vtx_data[labels == -1] = -1

brain.add_data(vtx_data, .5, .75, colormap="jet", alpha=.8)
brain.show()

[GuillaumeFavelier]

Most of the code comes from #6232 stripped from ipyvolume dependencies. What do you think @larsoner , @agramfort ?

[codecov]

Codecov Report

Merging #6460 into master will decrease coverage by 80.82%.
The diff coverage is 13.01%.

@@            Coverage Diff             @@
##           master   #6460       +/-   ##
==========================================
- Coverage   89.27%   8.44%   -80.83%     
==========================================
  Files         411     402        -9     
  Lines       74697   73403     -1294     
  Branches    12342   12248       -94     
==========================================
- Hits        66684    6200    -60484     
- Misses       5149   67148    +61999     
+ Partials     2864      55     -2809

[codecov]

Codecov Report

Merging #6460 into master will decrease coverage by 0.04%.
The diff coverage is 81.74%.

@@            Coverage Diff             @@
##           master    #6460      +/-   ##
==========================================
- Coverage   89.27%   89.22%   -0.05%     
==========================================
  Files         411      416       +5     
  Lines       74697    74371     -326     
  Branches    12342    12287      -55     
==========================================
- Hits        66684    66356     -328     
- Misses       5149     5154       +5     
+ Partials     2864     2861       -3

[GuillaumeFavelier]

Thanks to the discussion with the PyVista team, the latest fix allows us now to use the Brain object in a Jupyter Notebook interactively!

[agramfort]

nice ! it works with add_data etc?

…

[GuillaumeFavelier]

I can't upload a GIF right now but basically the three examples I shared in this thread works so basic usage of add_data() is indeed supported.

[larsoner]

How close are we then to having stc.plot() work?

Maybe the plan should be:

1. Review/merge this PR
2. Refactor stc.plot to use this new object
3. Add time_viewer=True support for each backend (where possible)

WDYT?

[GuillaumeFavelier]

How close are we then to having stc.plot() work?

Not everything is available yet but we can definitely add the missing features incrementally. I can also create a RST table to improve visibility on this work. Benchmarking this Brain object against the PySurfer example gallery will help as well.

Maybe the plan should be:

1. Review/merge this PR

2. Refactor `stc.plot` to use this new object

3. Add `time_viewer=True` support for each backend (where possible)

Sounds good to me!

[larsoner]

Benchmarking this Brain object against the PySurfer example gallery will help as well.

We probably do not need all of those examples, but we can go case by case. Let's aim to make sure stc.plot(...) works to start. Can I already try examples that use stc.plot and expect them to work with this PR?

Not everything is available yet but we can definitely add the missing features incrementally. I can also create a RST table to improve visibility on this work.

+1 for both.

[GuillaumeFavelier]

Can I already try examples that use stc.plot and expect them to work with this PR?

This PR adds the class Brain but stc.plot still uses PySurfer . I wanted to continue refactoring with another PR to be honest.

[larsoner]

Okay sounds good.

Running the code snippet from the top post with latest pyvista I see:

Scroll wheeling a lot to zoom out I get something more reasonable:

Do you see the same thing?

[GuillaumeFavelier]

Yes indeed!

[GuillaumeFavelier]

I can find a suitable value for the distance parameter in this PR if you want? It will fix the scrolling issue.

[larsoner]

You can just add it to a todo list in a follow-up PR. It will quickly become annoying to test stc.plot if this value is not set properly so I trust it will be fixed very soon :)

[larsoner]

One thing that I'd like to do for now, though, is make it _Brain and remove it from python_reference.rst.

The reason for this is that we might want to:

1. Rename it. Something like BrainSurfaces is more apt if we want to have volumetric and surface source spaces on the same footing.
2. Change the API. Right now it mimics PySurfer, which is nice to start. But this is our chance to fix things that annoy us about the API.

Okay with you to keep it private for a bit @GuillaumeFavelier ?

[GuillaumeFavelier]

Yes sure, I'll rename it _Brain for now.

[larsoner]

Thanks @GuillaumeFavelier, keep up the progress!