Jet-interpolation-grid evolution issues

[felixhekhorn]

In order to convolute jets we need to adjust the EKO side since we're limited by memory there (discussed in NNPDF/eko#138 and related issues/PRs)

I wonder, whether we're also limited by memory here, even with the new layout developed in #184. The actual size of the ekos is

$ du -ch 200-CMS_1JET_8TEV-CMS_1JET_8TEV-bin00*
[...]
554M	total

An easy solution, if we're limited, would be just to add a bin_mask argument to Grid::evolve (and one to Grid::axis, since we need to provide the correct ekos in the first place). In practice this was faked to compute theory 200 via the deviation of splitting the bins before computing the eko and rejoining them after the FK table computation.

[alecandido]

Have you tried to optimize the grid? Or is it that big after optimization?

[felixhekhorn]

it's the size of the ekos (which actually is not clear from the command line)

[alecandido]

it's the size of the ekos (which actually is not clear from the command line)

But if that is the full EKO for bin00, remember that there is the dynamic scale inside the bin. So, with the new output each individual operator (for each $\mu^2$ value) might be reasonable enough :)

[cschwan]

If you tell me how I should change these lines,

https://github.com/NNPDF/pineappl/pull/184/files#diff-7b7287991850a10d433203bd3a81118ecea8c293bfa5f3fde86451fe49eead99R2644-R2651

I can try it!

[felixhekhorn]

But if that is the full EKO for bin00, remember that there is the dynamic scale inside the bin. So, with the new output each individual operator (for each μ2 value) might be reasonable enough :)

exactly! they are ~2MB each - but this improve data structure can not be exposed to Grid::evolve

[felixhekhorn]

If you tell me how I should change these lines,

https://github.com/NNPDF/pineappl/pull/184/files#diff-7b7287991850a10d433203bd3a81118ecea8c293bfa5f3fde86451fe49eead99R2644-R2651

I can try it!

actually, how big is that eko? the problem I can see is passing around 500MB of data (and at some point we may have the double I guess, because Rust will copy from Python, I guess?)

[alecandido]

exactly! they are ~2MB each - but this improve data structure can not be exposed to Grid::evolve

Why not? Maybe not yet.

The final step of the new format was providing a Rust API to load it NNPDF/eko#97, and it should take too much (i.e. I believe I can take care of in a short time).

Since @cschwan already loaded successfully a Numpy array, I only need to unpack the tarball and do the same for the relevant operator.

[felixhekhorn]

exactly! they are ~2MB each - but this improve data structure can not be exposed to Grid::evolve

just to say that at some point we were actually thinking about exposing this data struture NNPDF/eko#97 but this is for sure mid-/long-term and not a short term solution to do NNPDF/pineko#62

[alecandido]

Actually, I consider that to be the short term one: the bottleneck is the computation of EKO in the new format, not reading it :)

[cschwan]

Could you provide me with a jet-EKO? I'd really like to test it.

[cschwan]

You can also try it yourself now, @felixhekhorn: in #184 I added a new subcommand evolve which should understand the EKO format:

pineappl evolve grid.pineappl.lz4 eko.tar fktable.lz4 PDFset

which evolves grid.pineappl.lz4 using eko.tar to produce fktable.lz4 and perform a check using PDFset.

[alecandido]

Could you provide me with a jet-EKO? I'd really like to test it.

Sorry for the no reply: problem is that we don't have. In order to evolve them, @felixhekhorn did the trick of splitting the grid by bin, evolve individually, and finally merge the FkTable.

This will be solved by the usual NNPDF/eko#138...

[alecandido]

You can also try it yourself now, @felixhekhorn: in #184 I added a new subcommand evolve

I saw: I believe we will always use the Python API through pineko.
But it is useful for quick checks, and it is also for external users: they may want FkTables, but pineko can still be to rigid for them :)