Ploughshare download

[andrpie]

Addresses #102 and NNPDF/pinecards#192.

This is the initial implementation of the class Plough that downloads the grids from ploughshare and converts them into pineappl format (or does whatever is necessary: cutting bins, renaming grids etc.)

Pinecard structure:

The folder would contain two files: ploughshare_link.txt and process_grids.sh (optionally postrun.sh and metadata.txt). Please have a look at the CMS_2JET_8TEV_3D example.
ploughshare_link.txt: contains the link to the file that has to be downloaded
process_grids.sh: responsible for conversion etc.

Class structure

Plough.run(): downloads the (tarball) file and extracts it in the output folder
Plough.generate_pineappl(): runs the process_grids.sh script inside the output folder (also tells it the grid names)

These methods are run when the class is initialised. If they were run conventionally (i.e. by run.py), then an external-pineappl comparison would have to be made at the pinefarm level - this is already done by pineappl import. Also, lines 194-200 of run.py assume that only one grid was generated.

The conversion happens inside process_grids.sh instead of postrun.sh, as postrun.sh only assumes that one grid was generated.

[felixhekhorn]

Please also run pre-commit.

[scarlehoff]

I agree with @felixhekhorn the run should be its own run

pinefarm/src/pinefarm/cli/run.py

Line 33 in 0490d3c

def subcommand(pinecard, theory_path, pdf, dry, finalize=None):

pinefarm will then run run and generate_pineappl (you can make generate_pineappl do nothing and put everything in a postrun.sh script that will be run as part of the postprocess method:

pinefarm/src/pinefarm/external/interface.py

Line 165 in 0490d3c

def postprocess(self):

which just runs postrun.sh and, importantly, burns in the metadata.txt into all grids which we want in general.

[andrpie]

Now Plough is compatible with postrun.sh and its run is not called in __init__. I've added a condition in run.py to make it work. The pinecard structure got updated accordingly. Thanks for the suggestions!

[scarlehoff]

Thanks!

Left a lot of nitpicks but seems to work (I tested CMS_2JET_8TEV_3D although I could not get grids because of Error: you need to install pineappl with feature fastnlo, which I won't; but I trust it would)

[scarlehoff]

Thanks for the changes! Once @felixhekhorn has time for a second look we can merge I think

[felixhekhorn]

As we discussed (when @scarlehoff was not around), I'd first like to have all relevant pinecards in NNPDF/pinecards#197 and NNPDF/pinecards#192 available, such that we are sure we have all tools and options we need in practice to discuss the different cases there (as the different datasets may need more or less structure)

[scarlehoff]

Is this already the case? I see many runcards there. Is that all?

[andrpie]

For single jets, I think there's one more runcard I should add. For dijets, I need to make the runcards compatible with the current shape of the Ploughshare module.

[andrpie]

The pinecards have been updated so that they are compatible with this module now:
dijets (fastNLO grids): NNPDF/pinecards#192
single jets (APPLgrid): NNPDF/pinecards#197