optional dependencies (matplotlib and scipy) are not truly optional

[rathann]

Expected behaviour

The lack of any of the optional dependencies listed in setup.py (netcdf4, matplotlib, scipy, seaborn and sklearn) is handled gracefully, with no exceptions thrown.

Actual behaviour

ImportError exception will be thrown when matplotlib or scipy are missing.

Code to reproduce the behaviour

The following imports don't catch ImportErrors when importing scipy:

MDAnalysis/analysis/psa.py:1936
MDAnalysis/analysis/psa.py:2053
MDAnalysis/analysis/polymer.py:169
MDAnalysis/analysis/pca.py:355
MDAnalysis/analysis/hbonds/hbond_autocorrel.py:443
MDAnalysis/visualization/streamlines_3D.py:42
MDAnalysis/visualization/streamlines_3D.py:43
MDAnalysis/visualization/streamlines.py:48

and matplotlib:

MDAnalysis/analysis/psa.py:1646
MDAnalysis/analysis/psa.py:1747
MDAnalysis/analysis/psa.py:1847
MDAnalysis/analysis/psa.py:1935
MDAnalysis/analysis/polymer.py:138
MDAnalysis/analysis/hole.py:372
MDAnalysis/analysis/hole.py:451
MDAnalysis/analysis/hole.py:520
MDAnalysis/analysis/legacy/x3dna.py:416

Currently version of MDAnalysis:

0.16.0

[kain88-de]

There isn't really a way for us to handle this more graceful then throwing an exception in those cases. For example in the pca case. If we can't import scipy that function can not return a suitable result because missing the integrator is a grave error justifying an exception.

How would you imagine here a graceful error handling?

[richardjgowers]

So something we could do here would be to do all the imports at the top of the file, then raise warnings about things that won't work. Eg in polymer, lots of it will work, but certain methods will need things, so a warning like ImportWarning: The following methods require scipy: PersistenceLength.curve_fit.

[rathann]

I do note that some of the missing optional dependencies are already handled nicely in the code. Thank you for that.

@kain88-de I'm just trying to express package dependencies correctly and let users uninstall dependencies listed in setup.py under extras_require. Having the software throw exceptions in case of missing optional dependencies is bad user experience and I'd like to avoid that. Perhaps some of these dependencies shouldn't be listed as optional, then.

@richardjgowers Yes, that'd be awesome.

[mnmelo]

So, this idea was already floated in #577, but we dropped when we realized it'd probably involve registering import hooks.

In the meantime I found the importing package, which has a lazyModule. Their implementation is hook-independent, and delays errors until the module is actually used.

I figure we can subclass lazyModule to customize error messages, to print something like "A functionality of MDAnalysis requiring module so-and-so was requested, but said module is not installed. Please install and retry.".

[mnmelo]

Using importing.lazyModule has the further benefit of not immediately loading potentially heavy optional dependencies (like scipy) until they're actually needed.

[orbeckst]

There's some discussion in #1159 , which we haven't really concluded. I think we are moving towards making more "core scientific python packages" (see #1159 (comment) for qualifiers of this term...) standard requirements of MDAnalysis.

[orbeckst]

Under this scheme, scipy and matplotlib would become required dependencies. I think, given Linux whl and conda I could live with this, especially, as these packages are only imported when a specific analysis module is imported; I therefore don't think that #1361 (comment)

not immediately loading potentially heavy optional dependencies (like scipy) until they're actually needed.

is a big issue. But if lazyModule works well, then we could try it – anything to make a user's life easier.

[orbeckst]

If we decide that scipy and matplotlib are required dependencies #1159 (comment) then we can close this issue. (We can then still look into other approaches like lazyModule but at least we would have solved this immediate problem.)