Split anomalous dimensions into a Rust crate

[alecandido]

The motivations for this issue are mainly:

1. we are using inconsistent tooling in the theory pineline to achieve performant computations: here we use Numba, while PineAPPL is implemented in Rust for the same reason
2. Numba has some limitations in terms of supported features, from this point of view Rust is much more flexible (it is a whole language with its own ecosystem, while Numba is bound to be a "subset" forever)
3. Numba seems to be especially slow at compiling N3LO terms
4. It is much more difficult to have bindings for anomalous dimensions for other languages, that might be relevant since we have original pieces

I'm currently trying to outline a suitable layout for the new Rust module, I'm doing some attempts in this repo:
https://github.com/AleCandido/atuin

I already asked for maturin authors support, but a minimal skeleton (no further crates for bindings) is almost working:
PyO3/maturin#1372

[alecandido]

The template works, as soon as we close the current main refactoring (i.e. #172) I will provide a PR with a working layout, but almost nothing in.

The proposal is to move the numba decorated content to a standalone crate (let's say ekspressions), while provide bindings to it in a separate crate.

In principle, we can also move the integration on the Rust side (such that all the heavy lift will happen there, and we do not cross the Rust-Python boundary during intensive operations). This I would do in the bindings crate, or a separate one, keeping ekspressions (stupid name, suggest a better one) for the analytical expressions alone.

To integrate in Rust we can use bindings to the GSL: https://docs.rs/GSL/latest/rgsl/integration/index.html (i.e. essentially the same as SciPy, but we have to install GSL ourselves).
Still looking for better options, but this discourages me to attempt to do it in the short term...

[felixhekhorn]

Compiling just took me about half an hour (!)

Matching: computing operators - 4/60 took: 1644.154465 s

I think we need to do something ...

[alecandido]

I think we need to do something ...

Agreed, and the first step is already in #189. But it is lower priority than other businesses, in particular FONLL.

However, if @giacomomagni wants to start having a look we can discuss. But for me and you it is out of limits, until everything else is working smoothly.

[alecandido]

Rust supports incremental compilation (by default only for debug, opt-in for production, reasonably):
https://nnethercote.github.io/perf-book/compile-times.html
https://doc.rust-lang.org/cargo/reference/profiles.html#incremental

[felixhekhorn]

@scarlehoff said in NNPDF/pineko#105 :

my computer crashed after eko took every possible resource,

and also discussing with @giacomomagni last week, we found the same conclusion: we should do something on the compilation front ...

[scarlehoff]

To be more precise, my problem is that I have many cores but not so much memory so my scratch got completely filled (quite rapidly).

[alecandido]

To be more precise, my problem is that I have many cores but not so much memory so my scratch got completely filled (quite rapidly).

You're not alone, it happened the same to me...

and also discussing with @giacomomagni last week, we found the same conclusion: we should do something on the compilation front ...

Yes, we should do something. But it won't be quick. The current workaround is to limit Numba processors for compilation, but it is taking a lot...