Memory saving output

[alecandido]

We realized that most eko operations do not depend on multiple Q2 at the same time.

The full OperatorGrid is a rank-5 tensor, by using only one Q2 at a time we reduce the problem to a rank-4 tensor, with much less memory consumption.

In order to get a usable and flexible structure, a few capabilities are needed for the new structure:

• separate Q2 storage: it will be {q2}.npy
• lazy loading: loads one (or a subset of) Q2 at a time, and drop them once consumed
  • even the subset might be useful, for cases in which the consumer only accepts a rank-5 tensor
• merge separately computed (but input compatible) outputs
• split a single output into multiple ones
  • not strictly needed, but it's dual to the former one, and so nice to have

This new structure will be the upgraded version of the current Output object.

Moreover, a few more things might be implemented as related, and later used to support separate computation of Q2 elements:

• replace/upgrade OperatorGrid
  • we need a manager for the computation, but it has not to hold the data, that has to be dumped as soon as possible
  • the easiest is to have the current OperatorGrid to hold the reference to a new Output object, and store everything in there
  • everything will include threshold operators, and partial Q2 results (those obtained before combining with threshold operators), together with full results
• support separately threshold operators and partial Q2 in Output
  • they are also dumped on disk, with their own names: thresholds.npz (containing all the threshold elements) and {q2}.part.npy

The idea is that, an object supporting these features, can be computed separately, in a completely independent way (1 process for thresholds, plus one for each Q2, for example), and then merged together.
In order to make it easier to merge and compute the final one:

• the thresholds.npz is never removed from the saved output
  • if other {q2}.part.npy come later, they can always be consumed
  • unless explicitly stated: provide an optimize() or clean() method, to get rid of it
• everything can be merged together, with or without thresholds, but it's simply checking the input compatibility, and adding the arrays to the archive
• if it contains both thresholds and partial objects, compute final ones
  • provide a combine() method
  • partial objects are removed after combination

[felixhekhorn]

separate Q2 storage: it will be {q2}.npy

at first I wanted to point out that we should use the byte representation, but then I thought maybe we could actually use this to introduce an approximation: we could use log(Q2) with a fixed precision (say 4 digits) and this way save some computations ...

merge separately computed (but input compatible) outputs

• before being able to merge full output objects, we first need to be able to extend existing outputs
• a simple condition for this is to dump the theory and operators card into the archive
• such that we can do Output.load("a.tar").get(2.)

support separately threshold operators and partial Q2 in Output

actually we could even join them only at loading time ...

they are also dumped on disk, with their own names: thresholds.npz

I'd rather put them in a separate sub-directory (just to keep them out of the way) - or we could even add them as regular operator, since after all they are regular operators leading to exactly the threshold (in the upper scheme)

[alecandido]

at first I wanted to point out that we should use the byte representation, but then I thought maybe we could actually use this to introduce an approximation: we could use log(Q2) with a fixed precision (say 4 digits) and this way save some computations ...

This is fine, but I wonder if Q2 might be simpler than log(Q2): I know the log is more relevant, but the other way it's easier to inspect manually, since you require fewer operations (but we can think about it).

• before being able to merge full output objects, we first need to be able to extend existing outputs

I was just thinking the other way round: the moment we can merge, we use this to extend.

• a simple condition for this is to dump the theory and operators card into the archive

I thought it was done, but maybe I'm only doing it for yadism...

actually we could even join them only at loading time ...

This I'm not sure: it's easier, because in order to extend you just need to drop more .npy files into the archive, but might be expensive. We can try to benchmark how much it takes, if it's negligible we can even do, but if the product is expensive, I would precompute it (or perhaps JIT, the first time you do it, you dump the product, unless you call .compile() or something like, and you do it AOT for all).

I'd rather put them in a separate sub-directory (just to keep them out of the way) - or we could even add them as regular operator, since after all they are regular operators leading to exactly the threshold (in the upper scheme)

I was thinking to store patches and matching separately, but maybe you're right and there is no purpose. If we store them as regular operators, we should mark them as thresholds in the metadata.

[alecandido]

we have a detailed plan for the output:

Plan

The folder will contain:

• metadata.yaml, containing output metadata (already present)
• runcards folder, containing all the runcards needed to reproduce the output
  • in case of products and later manipulations, all the involved runcards are saved to this folder, and the full history to reproduce is written in metadata.yaml
• recipes folder: this will contain brief YAML files with the recipes for jobs computing parts
• parts folder: containing all the partial outputs that have to be Mellin integrated
  • partial patches
  • matching conditions
• operators folder: containing the final results, after combining the parts
  • the files name will be binary representation of the Q2 floats + extension (all files here will be valid evolution operators)

[alecandido]

This is also more long term than #138 at this point: the core part of the memory structure has been done in #105, while in #138 the computation will be faced (that will include OperatorGrid removal and split threshold operators).

Every other feature here is considered a "nice to have", but no more.

[alecandido]

@felixhekhorn the part strictly addressing the title is already implemented, and most of the rest will be implemented as a consequence of #138

The only two elements that are falling outside #138 and contained here are:

• merge separately computed (but input compatible) outputs
• split a single output into multiple ones
  • not strictly needed, but it's dual to the former one, and so nice to have

Should we keep this issue for them?

[felixhekhorn]

I still consider the items relevant - maybe we can put them into a new issue (with a clearer title)?