Load modules lazily

[hakonanes]

Description of the change

This PR addresses #572 by loading modules lazily using lazy-loader. This becomes a new required dependency.

A new contributor guide briefly explains lazy loading and why we use it.

Running the following line, which requires the tuna package, python -X importtime -c "import orix.quaternion" 2> oqu.txt && tuna oqu.txt, produces the following browser output without lazy loading:

And with lazy loading (this PR):

We see a speed-up of roughly 33x for the quaternion module.

Other minor changes:

• Simplify listing of dependencies from a table to a list
• Update type hints in the contributor guide

Progress of the PR

• [n/a] Docstrings for all functions
• Unit tests with pytest for all lines
• Clean code style by running black via pre-commit

For reviewers

• The PR title is short, concise, and will make sense 1 year later.
• New functions are imported in corresponding __init__.py.
• New features, API changes, and deprecations are mentioned in the unreleased
  section in CHANGELOG.rst.
• Contributor(s) are listed correctly in __credits__ in orix/__init__.py and in
  .zenodo.json.

[CSSFrancis]

@hakonanes This is very nice, thanks for helping with this! Is orix.quaternion then compiled the first time that it is used?

I'm trying to remember exactly how that works. So something like this??

Edit I guess the numba code isn't really the slow part. It's just matplotlib.

from orix.quaternion import Rotation # this is now really fast

r = Rotation([1,2,3,4]) # this is now slow becuase the numba code is compiling 

r2 = Rotation([0,0,0,1])  # this is now fast

r1 *r2 # This is also fast

[CSSFrancis]

I guess my one gripe with the lazy loading is that it just feels like it's moving the blame around for who takes a long time to import. People used to get frusterated with how long it took hyperspy to load, when most of the time was matplotlib scipy etc. But all of those packages were using lazy loading so they weren't being blamed because importing them didn't take any time at all!

I guess the last package to not implement lazy loading gets stuck "holding the buck" so to speak.

Most of the time is spent loading orix.quaternion although I suspect that is just becuase it is "first" entry point because it is so widely used...

I guess my question is are we actually lazily loading anything or does almost every class import orix.quaternion and we are just making some other step slow.

The other question is does it make sense to make contributing to the package more difficult to just to make users feel better.

Just some thoughts. I like the idea of lazy loading, but sometimes I think it's almost like a shell game of hiding your imports.

[ericpre]

I guess my question is are we actually lazily loading anything or does almost every class import orix.quaternion and we are just making some other step slow.

The other question is does it make sense to make contributing to the package more difficult to just to make users feel better.

Just some thoughts. I like the idea of lazy loading, but sometimes I think it's almost like a shell game of hiding your imports.

I started to look at lazy loading in the last couple of days, because pyxem is slow to load, which is particularly inconvenient for testing code changes or running test suite, etc.!!! In my case, there are several reasons for it to be slow:

• having cupy installed slow down dask import significantly
• diffsims and orix are imported from the start and a lot of things can be done in pxyem without needing diffsims and orix.
• parsing the version with git information, slow down rosettasciio and hyperspy import... 🤦
• most likely MS Windows is to blame too... 😅

Generally, lazy import very useful, because it defers to import library for when they are necessary and quite often these libraries are not needed. Also, they are usually needed in functions which are not particularly fast.

[hakonanes]

Valid points, @CSSFrancis. Introducing lazy loading may not be revolutionary for the end user who uses functionality that uses most "heavy" modules. But as @ericpre points out, it would be a benefit to downstream packages, like pyxem.

does it make sense to make contributing to the package more difficult to just to make users feel better

Yes. Keeping the __init__.py clean and listing module imports in the module __init__.pyi file is argubaly not difficult to maintain. The argumentation should be clear from the contributor guide I've made. Please have a look at that one if you guys have any suggestions for improvements!

[CSSFrancis]

Some small comments otherwise this is good! I must say a lot of my hesistation is gone with the .pyi files. I hadn't realized that was another option for defining lazy imports. I don't know if it is less complicated than it was but it feels less complicated.

I think I'm convinced :) I'll try to make the changes to pyxem/didffsims as well which should help.

[CSSFrancis]

What is this used for?

[hakonanes]

Full project version: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-release

[CSSFrancis]

Maybe add something like:

Suggested change

	In the __init__.py file:
	.. code-block:: python
	import lazy_loader
	__getattr__, __dir__, __all__ = lazy_loader.attach_stub(__name__, __file__)
	del lazy_loader

It might also be good to explain how the init.pyi files should be set up. The getattr and dir are for attribute acess right and dir is for tab completion right?

[hakonanes]

I've added the following explanation:

The returns from lazy loader are:
- ``__getattr__``: function to access names defined by the module
- ``__dir__``: list of names a module defines
- ``__all__``: list of module, class, or function names that should be imported when
  ``from package import *`` is encountered

[hakonanes]

Thank you for the review, @CSSFrancis! I'll merge once tests pass.

[hakonanes]

Actually, before that, @argerlt, do you want to have a look?