ENH: Notebook 3d backend improvements (ipyvtklink)

[GuillaumeFavelier]

I open this issue following #8503 (comment) to keep track of the missing features or anything that could be improved.

Feature status

• Test integration with Google Colab following Using raw.plot in JupyterLab or Colab #8080 (comment)
• HiDPI rendering of 3D and mpl widget (see MRG, ENH: Add a screenshot button to the notebook 3d backend #8708 (comment))
• Screenshot w/time_viewer is off in the vertical direction (https://github.com/mne-tools/mne-python/pull/8708/files/5d34669c5ea051b1c08dfda19cb1ddf0ba727945#diff-a6e2fc2fab82c1377cb29516fc0a7aed813ee4273daad18cf002e7302a331770)
• Update to ipyvtklink (Rename to ipyvtklink Kitware/ipyvtklink#30, MNT: Migrate to ipyvtklink #9341)
• Button/shortcut for autoplay (ENH: Add a playback button to the notebook 3d backend #8741)
• Button/shortcut to take a screenshot (MRG, ENH: Add a screenshot button to the notebook 3d backend #8708)
• Button/shortcut to save a movie (ENH: button to save movie #9190)
• Button/shortcut to display help (ENH: Help button #9123)
• Label time course mode (MNT: notebook conditionals in Brain & _update #9016)

Design/UX

• big square button (asked in MRG: Prototype of notebook viz (ipyvtk) #8503 (comment))

---

Known issues

Resolved

• Does not play well with Firefox (see ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment) and ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment))

JupyterLab

• matplotlib is not interactive Javascript Error: IPython is not defined
• The mesh is translucent
• A VTK window opens on local instance

Google Colab

• It is not supported yet (asked in MRG: Prototype of notebook viz (ipyvtk) #8503 (comment), standby in ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment))

---

A testing notebook is available on gist.github.com.

[GuillaumeFavelier]

About Google Colab, I made some research @agramfort.

Considering that ipyvtk_simple's main viewer class, ViewInteractiveWidget, is based on ipycanvas's Canvas widget, I found that it is a known issue. The short answer is (and I quote Martin Renou):

Custom widgets libraries are not supported by Google colab yet

We could track the progress on this matter on https://github.com/martinRenou/ipycanvas/issues/170 or more directly in googlecolab/colabtools#498.

Anyway, I think those need to be resolved before more can be expected from ipyvtk_simple and finally mne-python.

[larsoner]

For movie and autoplay I think you basically need a timer, e.g. https://stackoverflow.com/questions/57255394/jupyter-and-timer-functions

For a screenshot it seems like it should be fairly easy since the ipyvtk_simple is sending bitmaps, and/or we should be able to get the Renderer directly and get the image from it

For label time course do you need more than https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html#Dropdown ?

For sizing I think it's https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Styling.html

[GuillaumeFavelier]

Exactly what I was looking for, thank you @larsoner 👍

[GuillaumeFavelier]

The mouse events on Firefox might be fixed by Kitware/ipyvtklink#25. I'll open a PR and give it a try.

[GuillaumeFavelier]

It might not be necessary, I don't have those issues anymore on Firefox 87.0