Structure code into submodules

[fredrik-johansson]

Currently pyflint.pyx includes all .pyx files textually. This is a headache for development: changing any one file and recompiling forces a recompile of all of python-flint, which takes forever. There should be separate submodules that can be compiled independently.

[GiacomoPope]

So, in #61 I did the absolute minimum to start exploring how we can start factoring this repository into submodules. The work so far has taken the base classes flint_* (e.g. flint_scalar) and put them into flint_base.flint_base. I have also taken the global context and placed this within flint_base.flint_context

For the remaining code, my thought was to do something like:

─ fmpz/
│  ├─ fmpz_types.pxd 
│  ├─ fmpz.pxd
│  ├─ fmpz.pyx
│  ├─ fmpz_vec.pxd
│  ├─ fmpz_vec.pyx
│  ├─ fmpz_mat.pxd
│  ├─ fmpz_mat.pyx
│  ├─ fmpz_poly.pxd
│  ├─ fmpz_poly.pyx
│  ├─ fmpz_mpoly.pxd
│  ├─ fmpz_mpoly.pyx
...

For each of the main types, then a user would do

from flint.fmpz import fmpz, fmpz_poly

The same would naturally extend to whatever functions we have.

[GiacomoPope]

However, if we're going through the work of refactoring and building up sensible class structures, then I think we might want something like

cdef flint_elem:
cdef flint_scalar(flint_elem):
cdef generic_ring(flint_scalar):
cdef fmpz(generic_ring):
cdef fmpq(generic_ring):
...

and so maybe a good first step would be to include the generic ring bindings into submodules in a clean and well documented way, and then we can refactor the existing code to sit on top of all these generics?

[oscarbenjamin]

For each of the main types, then a user would do

from flint.fmpz import fmpz, fmpz_poly

This still makes it impossible to import fmpz without loading fmpz_poly etc.

I think that it is better to just have precisely two ways of importing things:

Either use fully qualified imports so

from flint.fmpz.fmpz_poly import fmpz_poly

or we provide a convenience layer:

from flint.interactive import *

The codebase should only ever use fully qualified imports internally and each module should only import the things that it needs so that one import does not recursively import everything else.

I suggested having levels here:
#55 (comment)

Probably all "level 1" things like these should be under a particular submodule like flint.types or something and then levels 0 and 2 could have separate submodules.

Some Python tooling (Sphinx, mypy etc) struggles with the idea that a module has the same name as something inside the module like:

from flint.fmpz.fmpz import fmpz

It is especially problematic if the flint/fmpz/__init__.py does from .fmpz import fmpz because it creates an ambiguity about whether flint.fmpz.fmpz means the module or the type. I suggest just systematically adding something like _type to the name so then it could just be something like:

from flint.types.fmpz_type import fmpz

It is a bit redundant but that is not a problem for the internals of the codebase as long as the import statements are systematic and predictable. It is also fine to have long import statements like this for downstream codebases as long as module names are stable over time so it is better to make sure that you would never regret putting something at top-level in future.

For interactive use it is not nice to have to fully qualify the imports, hence flint.interactive (perhaps a better name can be used).

[GiacomoPope]

OK -- this looks sensible to me. I personally will be offline for a few weeks, so will be delayed in any work on this but my rough plan would be:

1. First remove all includes from the main file
   1. Create a corresponding .pxd for each .pyx for each current type. This will allow use to lighten _flint too, as we can have the type specific header info in these .pxd files.
   2. Add these files to the dir. types with the following structure

   ─ types/
   │  ├─ flint_types.pxd
   │  ├─ fmpz_type/
   │  │  ├─ fmpz.pxd
   │  │  ├─ fmpz.pyx
   │  ├─ fmpz_vec_type/
   │  │  ├─ fmpz_vec.pxd
   │  │  ├─ fmpz_vec.pyx
   │  ├─ fmpz_mat_type/
   │  │  ├─ fmpz_mat.pxd
   │  |  ├─ fmpz_mat.pyx
   │  ├─ fmpz_poly_type/
   │  │  ├─ fmpz_poly.pxd
   │  │  ├─ fmpz_poly.pyx
   │  ├─ fmpz_mpoly_type/
   │  │  ├─ fmpz_mpoly.pxd
   │  │  ├─ fmpz_mpoly.pyx
   │  ├─ fmpq_type/
   │  │  ├─ ...
   │  ├─ nnmod_type/
   │  │  ├─ ...
   

   3. Import all of these back into pyflint.pxd so that all current tests still work

Then, after this has been done, the PR can be modified so that the imports throughout the module work nicely at the python level?

[oscarbenjamin]

2. Add these files to the dir. types with the following structure

─ types/
│  ├─ fmpz_type/
│  │  ├─ fmpz.pxd
│  │  ├─ fmpz.pyx
│  ├─ fmpz_vec_type/
│  │  ├─ fmpz_vec.pxd
│  │  ├─ fmpz_vec.pyx

Would this create packages i.e with __init__.py for each of fmpz_type etc (not sure how this works in Cython)?

Also would it mean that there is an fmpz module with the same name as the type it contains e.g.:

from flint.types.fmpz_type.fmpz import fmpz

Perhaps simpler would just be

flint/
├─ types/
│  ├─ __init__.py
│  ├─ fmpz_type.pxd
│  ├─ fmpz_type.pyx
│  ├─ fmpz_vec_type.pxd
│  ├─ fmpz_vec_type.pyx

Then each .pyx becomes a runtime module in the types package and it is:

from flint.types.fmpz_type import fmpz

I'm not especially tied to any particular names here like types or _type but I do think that:

• Everything should be in a subpackage analogous to types/ above (this is needed for forward compatibility in case any redesign of the public import interface might ever be wanted in future).
• The module names should be different from the types (although in a predictable way).
• The general naming scheme and structure should be simple and predictable both for source layout and for knowing how to write the import statement.
• The overall structure should make it (at least in principle) possible that you could import only a subset of things without loading every module into memory at runtime.

[GiacomoPope]

I think your suggestion is better and agree with your bullet points.

After the refactor, I think what we can do in pyflint.pyx is

from flint.types.fmpz_type cimport fmpz
fmpz = fmpz

and then the .pxd will export it as it currently is done. Of course, we then want to do this

from flint.interactive import *

Or

from flint.types import fmpz

in the python world (or something similar) but this will be solved as we go?

[oscarbenjamin]

f course, we then want to do this

from flint.interactive import *

That would be my suggestion. Again I'm not tied to "interactive" as a name. Some people would certainly want to use it in a non-interactive context.

Ideally it would be all like:

from flint.all import *

However "all" is a builtin in Python so that name is best avoided.

Or

from flint.types import fmpz

It would have to be

from flint.types.fmpz_type import fmpz

because if the flint/types/__init__.py imports fmpz etc from each submodule then it again becomes impossible to import anything without loading everything into memory.

Even if we don't want to make a change to from flint import * yet I think that it is important to verify that the structure would make it possible to avoid loading everything into memory so that change should be tested (and memory use etc checked) before committing to any structure now.

[GiacomoPope]

In the context of #59 I'm back from my holiday and so can assist with the refactoring of the current code if this would be helpful. Once we have all come to a solid idea of a plan we should be able to all work together to make quicker work of the refactoring

[deinst]

If you could look at my fork and make any suggestions, they would be appreciated. I know the structure is all wrong and will spend some time today to rectify that by moving everything under a types subdirectory.

[deinst]

Now that I have caffinated, there are two simple questions that I basically punted.

1. Where do we store the ctx object? flint_base.flint_context seems like the best place. Why do ctx and thectx both exist?
2. What to do with FMPZ_UNKNOWN, FMPZ_REF, and FMPZ_TMP. It seems best to just cdef them as constants in _flint.pxd