Dump FK Table comparison

[felixhekhorn]

• rename metadata results to results_grid
• add metadata results_fk with comparison w/ against w/o evolution

will need NNPDF/pineappl#74 to read metadata

[cschwan]

* rename metadata `results` to  `results_grid`

What's the idea here?

[felixhekhorn]

we can also just leave it - but then there will be results and results_fk so it's a bit inconsistent (since now there are two results ...)

[alecandido]

What's the idea here?

The point is that in rr we store the comparison at the grid level, that is the main and only comparison there, but when you come to pineko the new one is the FkTable.

For sure we don't want to change rr because there will pineappl at some point, since rr is upstream, and independent of what happens afterwards. Then it makes sense to change it afterwards.

[cschwan]

we can also just leave it - but then there will be results and results_fk so it's a bit inconsistent (since now there are two results ...)

I'd argue for that, because results are the true results (remember there are two entries MC and PineAPPL), and results_fk are the results with errors from the evolution. It also makes sense when you compare a grid at the process scale and an FK table; both results should be the same. And in the FK table there should be both, of course.

Moreover, results_fk should show a different table:

• scale variations, which should be shown in results don't make any sense in results_fk
• instead you should compare pre- and post-evolution results and just show the difference in per mille.

So the table should look as follows:

------------------------------------
    Pineko       PineAPPL   central
                             1/1000
------------------------------------
 7.218832e-01  7.218831e-01   0.0000
 7.142003e-01  7.142003e-01   0.0000
 7.928799e-01  7.928799e-01   0.0000
 7.850020e-01  7.850020e-01   0.0000
 7.728221e-01  7.728221e-01   0.0000
 8.654099e-01  8.654099e-01   0.0000

[alecandido]

Fine for me, I was thinking that in some sense results is the main focus, and it is a bit weird for the FkTable. Nevertheless, we can keep also as it is, it's not going to hurt anything.

[felixhekhorn]

So the table should look as follows:

and in fact, it does (quasi):

$ python -mpineko compare data/mydis.pineappl data/myfktable.mydis.pineappl.lz4  CT14llo_NF4
LHAPDF 6.3.0 loading /home/felix/local/share/LHAPDF/CT14llo_NF4/CT14llo_NF4_0000.dat
CT14llo_NF4 PDF set, member #0, version 1; LHAPDF ID = 13207
       O0 left    O0 right       O1 left      O1 right   PineAPPL    FkTable  percent_error
0    50.000000   50.000000  2.000000e-07  2.000000e-07  12.826575  12.826106      -0.003654
1    50.000000   50.000000  6.984209e-07  6.984209e-07   9.550602   9.549435      -0.012216
2    50.000000   50.000000  2.438943e-06  2.438943e-06   7.047279   7.046536      -0.010548
3    50.000000   50.000000  8.516807e-06  8.516807e-06   5.152585   5.152081      -0.009781
4    50.000000   50.000000  2.973850e-05  2.973850e-05   3.728779   3.728443      -0.008991
5    50.000000   50.000000  1.038117e-04  1.038117e-04   2.665475   2.665247      -0.008558
6    50.000000   50.000000  3.620545e-04  3.620545e-04   1.874623   1.874467      -0.008317
7    50.000000   50.000000  1.258680e-03  1.258680e-03   1.290291   1.290190      -0.007889
8    50.000000   50.000000  4.328501e-03  4.328501e-03   0.875369   0.875307      -0.007070
9    50.000000   50.000000  1.437507e-02  1.437507e-02   0.615639   0.615608      -0.005093
10   50.000000   50.000000  4.341492e-02  4.341492e-02   0.463775   0.463782       0.001509
11   50.000000   50.000000  1.091438e-01  1.091438e-01   0.361635   0.361633      -0.000531
12   50.000000   50.000000  2.195041e-01  2.195041e-01   0.288607   0.288606      -0.000214
13   50.000000   50.000000  3.668753e-01  3.668753e-01   0.195669   0.195669       0.000099
14   50.000000   50.000000  5.397572e-01  5.397572e-01   0.089400   0.089399      -0.001851
15   50.000000   50.000000  7.295868e-01  7.295868e-01   0.018904   0.018904       0.003388
16   50.000000   50.000000  9.309441e-01  9.309441e-01   0.000235   0.000236       0.515855
17   46.538363   46.538363  1.091438e-01  1.091438e-01   0.361349   0.361348      -0.000401
18   85.950595   85.950595  1.091438e-01  1.091438e-01   0.363406   0.363405      -0.000486
19  158.740105  158.740105  1.091438e-01  1.091438e-01   0.364716   0.364714      -0.000363

Note that the bin informations are also present here - something I also wish for results because otherwise I can't compare to the data from @scarlehoff who does apply cuts, and thus, e.g., presents only numbers for bin 6 onwards ...

[cschwan]

This looks good! However, I'd suggest to change a few details, namely

• to print the differences in per mille instead of in per cent
• instead of O0 left and O0 right you could print
  • the actual observable names that are in the metadata,
  • and print the observable names for both columns and leave out left and right

[cschwan]

@alecandido @felixhekhorn another completely unrelated question that also came up during the PC: in DIS the left and right bin boundaries are the same, but I wonder is this really the case also in the data? Don't we have bins in this case?

[alecandido]

• the actual observable names that are in the metadata,

unfortunately to do this from python we still need to merge NNPDF/pineappl#74 (we do not have access in the python interface to metadata, but we are going to have)

@alecandido @felixhekhorn another completely unrelated question that also came up during the PC: in DIS the left and right bin boundaries are the same, but I wonder is this really the case also in the data? Don't we have bins in this case?

This I don't know, we should check commondata, with which I'm not very familiar... (yet)

[cschwan]

• the actual observable names that are in the metadata,

unfortunately to do this from python we still need to merge N3PDF/pineappl#74 (we do not have access in the python interface to metadata, but we are going to have)

I see, I'll prioritize this Issue now.

[felixhekhorn]

in DIS the left and right bin boundaries are the same, but I wonder is this really the case also in the data? Don't we have bins in this case?

I don't think so - at least for the HERA fully inclusive stuff: https://www.hepdata.net/record/ins1377206?version=1&table=Table%202 (I didn't check the paper fully though ... and I know in practice this is rather required)

[cschwan]

Given what we've discussed last PC, I think we want the following information in the metadata that's not yet in there:

• MC uncertainty
• total error (difference between MC result and FK table result using the same PDF). This might require a standardized form for the results metadata and a new results_pdfset entry

[alecandido]

• total error (difference between MC result and FK table result using the same PDF). This might require a standardized form for the results metadata and a new results_pdfset entry

If we're going to have a new standardized form results entry, I would make it in whatever standard format (even csv, with whatever sep) in order to make as easy as possible to machine parse it.
Now they are more optimized for human reading, but in case we need to read them frequently I would make a printer for it. In case it is more generally needed, I'd include the printer in pineappl_cli.

[cschwan]

@alecandido agreed, but I think we can have it both machine and human readable while only deviating a little from what we already do:

• table headers get prefixed with "# " (comment character)
• content left as is

The resulting table should be easily readable with numpy.loadtxt.

[alecandido]

@felixhekhorn if you can, try to address this as well during your pineko transition.

Since you're moving comparison from fkutils (if it were not already here), now the information should be available locally, so it's just a matter of dumping it in the metadata during convolution.