[WIP] unlock internal effective exponents functions

[wilsonmr]

as per title

closes #683

I want to check this a bit more thoroughly but now one can plot efffective exponents specifying just PDFs and their own basis and flavours:

template_text: |
 # Effective Exponents in Flavour Basis.
 ## Internal function
 {@ plot_betaEff_internal @}

basis: flavour
pdfs:
 - NNPDF31_nnlo_as_0118_1000

actions_:
 - report(main=True)

or they should be able to do the old thing as before:

template_text: |
 # Effective Exponents in Flavour Basis.
 ## Internal function
 {@ plot_betaEff @}

fits:
 - NNPDF31_nnlo_as_0118_1000

actions_:
 - report(main=True)

Likewise the effective_exponents_table_internal should work to some extent. You will notice that if no fit is provided from which to take "previous exponents" then they are set to NaNs so that they are not plotted here is a quick example using a random PDF I had installed:

https://vp.nnpdf.science/DZ2UV-0vR3eZtwqY0aAzNg==

clearly todo:

• Improve the index of the eff exp table to use nicer labels
• check that the old behaviour is reproduced
• introduce some tests..
• improve documentation so it is clear which actions to use

[wilsonmr]

there's probably a nicer way to do this so that we don't have to check for this condition twice..

[wilsonmr]

@rabah-khalek in the end I left the conditions as are so basically we do something special for gluon and singlet and then the same thing for everything else. Obviously this could be reviewed but as a jumping off point it didn't seem so unreasonable

[Zaharid]

Would it make sense to have something that knows nothing of previous exponents, something that only knows about previous exponents and something that puts them together when needed (in the correct basis)?

[Zaharid]

Also it seems to me that the special conditions belong in pdfbases.py (i am now remembering this came up but never got done).

[Zaharid]

Would it make sense to have something that knows nothing of previous exponents, something that only knows about previous exponents and something that puts them together when needed (in the correct basis)?

So to elaborate:
Action1: Compute effective exponents anew.
Action2: Read limiting exponents from the fit.
Action3: Make sure action 1 is in the same basis, and put together action1 and action2 in one table.

[wilsonmr]

Would it make sense to have something that knows nothing of previous exponents, something that only knows about previous exponents and something that puts them together when needed (in the correct basis)?

probably, sorry for some reason I didn't get any notifications about your comments

[wilsonmr]

ok I split up the action into two, it's a bit neater now. I haven't yet moved special conditions just because I wasn't sure how much to move, is it the special case for gluon and singlet? If so then I don't see how that is basis dependent, in the sense that gluon appears in both flavour and fit basis and surely wants the same rule applied in both?

[Zaharid]

If I was doing it today, I would have the concept of flavour as a separate entity, with basis a lightweight wrapper over collections of flavours (note that probably can be done without changing any code outside pdfbases). Like it is now, you have to add the thing to all the bases that use 'gluon'.

That said, I am fine with doing this "later".

[Zaharid]

Hello, this is @Zaharid's automated QA script. Please note it is highly experimental. I ran pylint on your changes and found some new issues.

On validphys2/src/validphys/eff_exponents.py, pylint has reported the following new issues:

• Line 288: Too many local variables (31/22)

[Zaharid]

This seems to be working for me.

[scarlehoff]

Many thanks,

Before merging I'd like to have some documentation on how to use it/what can be used/etc. Otherwise in 4 years when we are working with NNPDF 5.0 someone would be lost again for one week wondering what was done by the ancient developers of NNPDF 4.0 :P

I'll do what we talked about this morning and then I'll be able to commit also some docs adding reports and information.

[Zaharid]

@scarlehoff As for right now there are some examples in the vp example directory. But those are generally not well maintained and I agree we want real documentation.

[wilsonmr]

Yeah, also there are new actions, for your purposes I'd initially run next_effective_exponents_table in the flavour basis, I suppose you could also use the plot_*Eff_internal if you want as well, for this you need pdfs as opposed to pdf but this can be a list of a single pdf

[scarlehoff]

Quick fit using the information obtained by this PR

https://vp.nnpdf.science/y2zuydh4T0ym38-bkciJfQ==

Many replicas are not that fantastic but it is a good start.

[siranipour]

I find the dip in the gluon interesting

[scarlehoff]

I put in a bit more flexibility to the previous fit as I felt it was what was missing in the previous one, https://vp.nnpdf.science/L33wX9PlSQWGoZkBrpnfjg==
Bottom line is, it is possible to do a fit in the flavour basis that looks very much like the old ones using as a starting point the preprocessing ranges from this PR.

The choice of preprocessing might still not be perfect but it's a good baseline. Next week I'll add a PR with the n3fit-flavour basis and some docs for this one so that someone can follow up.

cc: @juanrojochacon @stefanoforte

[Zaharid]

What's up with the uncertainties, e.g. https://vp.nnpdf.science/L33wX9PlSQWGoZkBrpnfjg==/figures/pdf_report_pdfnormalize0_basespecs1_pdfscalespecs0_plot_pdfs_g.pdf

anyhow, let's discuss that somewhere else.

@wilsonmr should we merge this?

[wilsonmr]

yeah let's go for it, can add docs later

[stefanoforte]

not sure I understand what has changed: it looks like now the small x exponents are still weird, but somehow the funny behaviour only sets in at extremely small x so everything looks alright. Anyway, it looks good! Juacrumar writes: I put in a bit more flexibility to the previous fit as I felt it was what was missing in the previous one, https://vp.nnpdf.science/L33wX9PlSQWGoZkBrpnfjg== Bottom line is, it is possible to do a fit in the flavour basis that looks very much like the old ones using as a starting point the preprocessing ranges from this PR. The choice of preprocessing might still not be perfect but it's a good baseline. Next week I'll add a PR with the n3fit-flavour basis and some docs for this one so that someone can follow up. cc: @juanrojochacon @stefanoforte — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

…

--

[juanrojochacon]

Hi @scarlehoff , sorry I forgot to reply to this one. The fit looks much better know, though I am not sure of what changed? Is only the range of preprocessing exponents? As @Zaharid mentions there are still rather large differences at the level of PDF uncertainties, but perhaps once one iterates preprocessing things will converge and agree better.
So this points into the right direction, which is most reassuring. I note that some flavours at large-x can still be quite negative, so imposing the new positivity criterion will change things quite a bit. Also, we know that DIS-only fits are good only for quick tests but that differences will be much smaller at the level of the global dataset, since having much more data leaves little wiggle room

[juanrojochacon]

So I guess that all looks good but having a global fit in the flavour basis would be the really crucial check to make sure there are no unexpected problems