n3fit closure tests

[wilsonmr]

Fits to be ran

DIS:

• 191007-tg-001-31dis TG
• 191007-mw-001-n3fitdis MW/TG

Global:

• 191007-ern-001-31glob ERN
• 191007-mw-002-n3fitglob MW

I have prepared the runcards for aforementioned fits, they're in the table at the top of https://www.wiki.ed.ac.uk/display/nnpdfwiki/n3fit+closure+results

NOTE: For the global runcards one will notice I have gone with the 3.1 configuration as opposed to the alphaS settings (Jets have sys 10 but no cfac: QCD) I think this makes sense from the perspective of isolating a change in the theory to a change in the methodology but if somebody feels strongly the other way then perhaps we should change this

---

Ok I thought it would be useful to open an issue

As it stands there is #514 which concerns running closure tests in n3fit. My personal opinion is that without #476 that this is the most reliable way to run closure tests for the time being so I think getting this merged or in a state where it can be merged ASAP should take priority

Next there is analysing the closure tests #511, this too is pretty close to being merged. Some documentation needs to be added either in #511 or #514 on running and analysing closure tests

[scarlehoff]

Then all there is to do is to run closure tests so that we have some results for the phone conference. I might need some help running a proper baseline closure test with nnfit, instead of taking as baseline the n3fit methodology which I am supposed to be testing

I am not sure you can get any baseline beyond "same sets of experiments", no?

I have never run a closure test myself but I would say a closure test can be run taking as a baseline the DIS runcard of the n3fit repository (real underlying PDF being for instance NNPDF 3.1) and the branch of PR #514. This will give a minimal working example and will make obvious if there is any issues with PR #514.

With regards to documentation I think #514 should have the documentation on how to run the closure test (personally I am ok with a bash script kind of guide) and #511 should have the documentation on how to do the analysis.

[wilsonmr]

I am not sure you can get any baseline beyond "same sets of experiments", no?

pretty much, plus same seed for the level 1 generation, that's all I meant :) the thing is people will want to see closure test estimators vs. using old methodology with otherwise identical settings. Which is something I can't currently plot. When I say I might need help I mean that the edinburgh cluster is fine for running DIS only nnfit and either DIS or global n3fit, but if I run a global nnfit then it will take over a week because of my low queue priority

I have never run a closure test myself but I would say a closure test can be run taking as a baseline the DIS runcard of the n3fit repository (real underlying PDF being for instance NNPDF 3.1) and the branch of PR #514. This will give a minimal working example and will make obvious if there is any issues with PR #514.

Yep, this was basically what I presented at Varenna except using MSTW as the underlying for basically no reason. The point is you first can check that it reproduces the underlying PDF in some way but then you can also compare to the old methodology which is fitting the same underlying PDF

With regards to documentation I think #514 should have the documentation on how to run the closure test (personally I am ok with a bash script kind of guide) and #511 should have the documentation on how to do the analysis.

Ya there are some things which in general need to be written down, because there are in fact quite a few closure test parameters which are AFAIK undocumented

Same with analysis, in practice it's not difficult but should be written down somewhere

[scarlehoff]

[...] the thing is people will want to see closure test estimators vs. using old methodology with otherwise identical settings. Which is something I can't currently plot.

You can't currently plot because of resources? If so I would say the comparison with the old methodology could be (for now) be restricted to DIS.

Yep, this was basically what I presented at Varenna except using MSTW as the underlying for basically no reason. The point is you first can check that it reproduces the underlying PDF in some way but then you can also compare to the old methodology which is fitting the same underlying PDF

Same as before, I think we can do that at DIS only level and, if/when needed we can get the resources for going global.

[wilsonmr]

of course, this issue was a long medium term to do list. One of the TODOs is to actually run the baseline fits. The global baselines essentially need to be ran elsewhere

I was just making a note of that so I can refer back to this and even tag people if I needed them, for example if I want somebody to run a global fit on hydra..

[wilsonmr]

@enocera I have tentatively assigned you to run the global baseline closure (with nnfit) do you have the cluster time available to run this fit?

[wilsonmr]

@tgiani as discussed if you could run the DIS baseline that would be much appreciated, I will try and run both the n3fit closures but will let you know if I need additional help with this

[wilsonmr]

I should also say: for the baseline fits there is no need to be on a particular branch in #514 I onyl make changes to n3fit and so either a conda install or master development build will be fine

[enocera]

@enocera I have tentatively assigned you to run the global baseline closure (with nnfit) do you have the cluster time available to run this fit?

Yes, I do.

[wilsonmr]

Great! Would it be possible for you to run that fit some time this week?

[enocera]

I'll run it right now.

[wilsonmr]

Thanks! Much appreciated