add final covmat regularisation procedure and a stat estimator table

[wilsonmr]

added the final recommendation for covmat reg to calcutils, added functions which regularize both covmat and sqrt of covmat (lower triangle/systematics file)

Also added a preliminary table for comparing the affect of different thresholds on the difference between the chi2 and the expected chi2 in units of the standard deviation of the chi2.

To do:

• improve table
• potentially add more tables
• check documentation, I copied a lot of the code from https://github.com/NNPDF/nnpdf/tree/regfrol2 so it might be wrong

closes #584

[wilsonmr]

The purpose of this was to have stable versions of the functions which regularize the covmats/sqrtcovmats on master and also some of the tables for the paper can perhaps be generated with validphys more efficiently than in a notebook.

[wilsonmr]

example of two tables added: https://vp.nnpdf.science/EJsoHwoDQeyfWe0l3bAW6g==

[Zaharid]

I have a couple of questions regarding the interface. I have the feeling that it is better the way it is done already though.

Now we have an opt in boolean flag that enables the regularization and additionally a threshold number. Could we have instead threshold=Optional[Number] with the meaning that there is no regularization if it is None?

Also would it make sense if the regularized covmat is some sort of explicit node instead of piling up functionality on top of the function we have? My intuition is no because we don't have some diverging dependencies for the normal and regularized mat (and you would still have to call it anyway).

More importantly how would this interact with the fit? Ideally there should be an easy way (for n3fit at least) to do fits with regularized covmats and to get the regularization from fit somehow. Possibly by adding the setting to the various fitcontext namespaces.

[wilsonmr]

Now we have an opt in boolean flag that enables the regularization and additionally a threshold number. Could we have instead threshold=Optional[Number] with the meaning that there is no regularization if it is None?

This seems sensible, changed to reflect this

Also would it make sense if the regularized covmat is some sort of explicit node instead of piling up functionality on top of the function we have? My intuition is no because we don't have some diverging dependencies for the normal and regularized mat (and you would still have to call it anyway).

I've never really looked at the explicit node stuff, I think the way it's currently done seems fine - like you say the normal and regularized covmats are used in the same way

More importantly how would this interact with the fit? Ideally there should be an easy way (for n3fit at least) to do fits with regularized covmats and to get the regularization from fit somehow. Possibly by adding the setting to the various fitcontext namespaces.

hmm more complicated because what this PR does not do yet is the correct regularization on experiment covmats, since we really need to commondata file to do that it was more for the dataset chi2s. I suspect this feature will be easier to do with the experiment API, because we can regularize the sqrt cov with regularize_l2 function before any 'group of datasets' covmat is constructed - if at all.

[wilsonmr]

hopefully tests pass then this closes #584

[Zaharid]

Cool.