Replace `Grid::axes` with `Grid::evolve_info`

[cschwan]

Grid::axes is too restrictive, see the discussion here: NNPDF/pineappl#197. Therefore it should be replaced with Grid::evolve_info.

[scarlehoff]

axes and evolve_info do not produce exactly the same xgrid in all scenarios.

In order to test #84, I'm regenerating fktables for the whole of theory 400. Doing that I've discovered that for some grids (more explicitly, for ATLASDY2D8TEV-aMCfast_obs_1.pineappl.lz4, note that obs_0 is ok!) evolve_info and axes return grids of different sizes.

This is independent of the orders selected when asking for evolve_info.

Inspecting the two x-grids returned, axes and evolve_info differ because the grid returned by axes repeats two values:
0.00094688 and 0.00335233. I'm assuming evolve_info is rightly sanitising the results while axes is not. All checks of pineko pass with flying colors (because the content of xs in the grid and the operator is indeed the same). However, during the evolution, pineappl says "no way Jose".

I'm copying the error below although now I think that maybe it is a pienappl bug (why are evolve_info and evolve using a different grid?) or a grid problem (why does the grid repeat one value of x?) or both... @cschwan @alecandido @felixhekhorn

The grid is here. The error occurs independently of whether the eko was computed using main or #84 (so, it occurs independently of how the eko was computed, since the incompatibility seems to be introduced here, so using the xgrid from axes works but from evolve_info doesn't).

Python error when using pineko:

  File "/media/juacrumar/pinekoing/pineko/src/pineko/evolve.py", line 209, in evolve_grid                                                                                                                             
    fktable = grid.evolve(                           
  File "/media/juacrumar/pinekoing/pinevortex/lib/python3.10/site-packages/pineappl/grid.py", line 338, in evolve                                                                                                     
    self.raw.evolve(                                 
pyo3_runtime.PanicException: Nothing returned from evolution.: EvolutionFailure("operator information does not match grid's x-grid values")  

I've tried using the cli but I get the following error independently on which branch I used to compute the eko.

Error: missing field `Q2grid`

[alecandido]

Inspecting the two x-grids returned, axes and evolve_info differ because the grid returned by axes repeats two values:
0.00094688 and 0.00335233. I'm assuming evolve_info is rightly sanitising the results while axes is not.

In principle, the deduplication is almost the same:
https://github.com/NNPDF/pineappl/blob/f2152da2a6b0c387b6b4c999b9025489dc02c580/pineappl/src/grid.rs#L1808-L1809
https://github.com/NNPDF/pineappl/blob/f2152da2a6b0c387b6b4c999b9025489dc02c580/pineappl/src/grid.rs#L1353-L1354

The only differences are in the sorting step:

# .axes
x_grid.sort_by(|a, b| a.partial_cmp(b).unwrap());
# .evolve_info
x1.sort_by(f64::total_cmp);

and that axes is deduplicated only just before returning, while evolve_info is doing it for each iterated subgrid.

The other major difference is related to your original issue about Grid::axes: that is checking all x-grids to be the same for one (order, bin), and concatenating across different (order, bin), while Grid::evolve_info is just concatenating them all.

[felixhekhorn]

I've tried using the cli but I get the following error independently on which branch I used to compute the eko.

Error: missing field `Q2grid`

which CLI do you mean? eko? do you have more of the environment (e.g. exact call and full error)?

[scarlehoff]

pineappl evolve grid.lz4 eko.tar nnpdf40

[cschwan]

It probably can't cope with the new/changed (?) EKO format.

[alecandido]

It probably can't cope with the new/changed (?) EKO format.

https://github.com/NNPDF/pineappl/blob/f2152da2a6b0c387b6b4c999b9025489dc02c580/pineappl_cli/src/evolve.rs#L31

Now called mu2grid, because Q2 was inconsistent...

[cschwan]

• @scarlehoff where can I get the EKO?
• @alecandido is the EKO format version documented somewhere?

[alecandido]

• @alecandido is the EKO format version documented somewhere?

Properly documented, no, I'm sorry.
We still need to make some small changes to account for the numerical FONLL project, and I'm currently updating the runcard format NNPDF/eko#220

However, in principle it is actually pretty simple, and those changes are going to have a minor impact, so I can write it here and actually add to the docs, even if it will be updated and improved:

• an EKO file is a tar file, e.g. eko.tar, with a specific internal structure
• the internal structure is currently dictated by a dedicated class in eko: eko.io.struct.InternalPaths, and it is:

  metadata.yaml
  theory.yaml
  operator.yaml
  recipes/
  parts/
  operators/
  

  this is pretty stable, and there is no foreseen change
• the user facing part is mostly metadata.yaml and operators/, both have an interface in EKO, specified by the dedicated classes (eko.io.struct.Metadata, and directly eko.io.struct.EKO for the second) - you might be interested in the original runcards, but they are there mainly for reference (usage required information is duplicated and possibly updated in metadata.yaml)
• the actual operators are stored in operators/ in .npy format (or .npz, in case they include integration error) as rank 4 arrays
• the exact name reflects the value of their final scale, but this is going to change (because numerical FONLL will require to specify both the scale and the number of flavors, so the scale alone will be ambiguous)

[scarlehoff]

In the atached .tar.gz you can find one grid, one eko and one script that does the evolve that pineko does. If inside the script you use the xgrid form axes instead of evolveinfo it works.

You can run it as python evolve_this.py ATLASDY2D8TEV-aMCfast_obs_1.pineappl.lz4 ATLASDY2D8TEV-aMCfast_obs_1_mainbranch.tar if you have eko and pineappl installed.

tmp.tar.gz

[cschwan]

I'm not sure what's going on here, but It seems the EKO is wrong. How can I see the process-level x-grid values of the operator? Is there an easy way to extract them from the tarball?

[scarlehoff]

I think you can obtain the grid you want doing:

np.exp(operators.rotations.targetgrid.grid)

At the end of the script in the tarball (note the np.exp since the grid you get already had a log applied to it). This grid corresponds to the grid that pineappl returns when you doevol_info.

I would argue that even if the eko is wrong, pineappl should not return two different grids with axes and evolve_info.

[alecandido]

Another undocumented feature: Xgrid.raw is a property showing exactly the raw grid, independently on being log or not.

Thus

operators.rotations.targetgrid.raw

is what you want.