Reproducibility of diagonal covmat fits (clarify or bug?)

[scarlehoff]

In principle, when savepseudodata is set to True (the default), the fit should be saving a csv file with the training and validation data per replica. This csv includes the name of the datasets and the index of the data.

When using the diagonal basis, there is no one-to-one correspondence between the datasets and the data going into training and validation, however these .csv file still have the experiment name on them.

We have pseudodata_test_fit_n3fit_251104 in the tests which is tested in test_read_matches_recreate so it seems we are able to recreate the pseudodata of a diagonal fit (separating training and validation). So, is this a bug? Or are we simply saving a dataset name that means nothing?

(or, rather, is the bug the fact that we are saving the name or are we saving something that is not really the data we are fitting?)

[achiefa]

I @scarlehoff , thanks for opening this. Indeed I had a look last week and the conclusion I reached is that we save data names that mean nothing. In fact, if we fit in the diagonal basis, the rotated data that enter the fit are combinations of the original data. This means that the same datapoints can be both in training as well as in validation. So saving the pseudodata using the dataset index is wrong. We should save using a combination index. Now, it might be that in this case is just the index to be wrong, while the actual values are correct. But this I can't really tell.

[jacoterh]

As far as I remember, @achiefa is right and the pseudodata that is saved corresponds to the original (data) basis even when using the diagonal basis. The reason being that we do not want to save the eigenvectors for the sake of storage. I will take a look and get back to you here!

[jacoterh]

I looked into this and I can confirm the pseudodata that we save for each replica corresponds to the original basis, with a mask corresponding to the diagonal basis. Given that the mask is random anyway, this second point is not an issue a per se, it's rather simply confusing that we store pseudodata in the original basis which does not correspond to the actual pseudodata that is subsequently used in the fit. The rotation happens after saving the pseudodata. I think we adopted this method given that rotated pseudodata is just as meaningless, and arguably even more, if one does not store the eigenvectors alongside them...

[achiefa]

Hi @jacoterh, thanks for having a look. Are you saying that we store the pseudodata before rotating in the diagonal basis, yet using the mask in the diagonal basis. So basically the stored pseudodata cannot be used to reproduce the fitted data, or am I wrong?

[scarlehoff]

So, just to understand, the data in the files nnfit/replica_X/<>.csv is then meaningless but the mask is correct. Is that it?

So in principle we have reproducibility as, given the runcard, one can:

1. Reconstruct the diagonalization
2. Read up the data and the mask from the .csv files
3. Rotate the data and apply the mask

We should add a disclaimer to the top of the files or something at the very least and ensure that the function that reads the pseudodata from the fit actually does that (which it doesn't at the moment but rather takes the pseudodata as is)

nnpdf/validphys2/src/validphys/pseudodata.py

Line 118 in b59080c

val = pseudodata[pseudodata["type"] == "validation"]

(that function takes as input the fit and the csv files so the information is all there, it will be much more expensive for the digonalized fits but so be it)

[achiefa]

Wouldn't it be better to store the data in the diagonal basis?

[scarlehoff]

Would be easier. I'm not sure why are we not doing that in the end. The eigenvectors need to be stored only once together with the covmat.

But in any case I think it is better to do one thing at a time, what I suggested should be fast and recover reproducibility as it doesn't depend on any random values (the random values were used for the mask and the replica and we have those).

Then we save the diagonalization and data in the diagonal basis... and if we do that I think we should drop the .csv files and use .npz to avoid the growth of the folder which is already quite heavy :__

[achiefa]

I still don't understand why we need to store the eigenvectors. These can be computed from the covmat, so if we store the latter we can always retrieve the eigenvectors. What we need to store is the mask, for each replica, in the diagonal basis. Then we apply the mask to the eigenvectors to select the combinations that were taken in trainining/validation. Is this right?

[jacoterh]

I agree we should just store the eigenvectors and write the pseudodata in the diagonal basis. This is clearly the cleanest solution, thanks for pointing it out! In principle we could reconstruct the eigenvectors on the fly, but given that these may not agree due to small numerical differences across different machines, we should still store it. @achiefa and I will coordinate this early next week!

[scarlehoff]

I still don't understand why we need to store the eigenvectors.

Same as with the data, so that in 4 years time we know what went into the fit.

(and tbh, it is something that is useful only once in a blye moon, but the time you need it it saves your life xD)