np.exp() makes libNNPDF fail down the line

[scarlehoff]

The test_rebuilddata.py test fails for python 3.9 in github actions.

• It doesn't fail in my own installation
• I've tried using a PC with Ubuntu to be a bit closer to the github action and it also works
• It used to be inconsistent but for Change ThPredictions to python predictions #1430 has failed 4 times in a row

https://github.com/NNPDF/nnpdf/runs/4324994519?check_suite_focus=true#step:6:2498

The only way of recreating the issue I can think of is using a docker image. I'll do it when I have some free time, if anyone has any clues or want to debug this feel free. (I would do it directly with #1430)

[Zaharid]

I wonder if that particular bug happens because the commondata files in the fit used for testing contain "nan" strings which are parsed in an undefined way, similar to #1465. That doesn't really explain why it happens intermittently or with the new code, but it may be worth checking.

Additionally, it would be nice if we could get better error messages out of the failure, e.g. by placing some explicit test here

nnpdf/validphys2/src/validphys/covmats.py

Line 391 in eeae19b

decomp = la.cholesky(correlation_matrix)

that tries to print out the index of the offending data on failure. @RoyStegeman mind having a look?

[RoyStegeman]

Sure, I can try

[Zaharid]

Trying to run that test locally I also see nothing resembling the issue on github.

[scarlehoff]

Trying to run that test locally I also see nothing resembling the issue on github.

The github servers are clearly in the upper atmosphere

[scarlehoff]

Ok, with the docker img I can get it to fail.

Turns out, test_rebuildata.py fails only when it is run after test_plost.py. Looking inside I can trace the problem down to

nnpdf/validphys2/src/validphys/plotoptions/core.py

Line 352 in eeae19b

return apply_to_all_columns(kintable, info.kinematics_override.xq2map)

And more specifically, the test fails due to this line here

nnpdf/validphys2/src/validphys/plotoptions/kintransforms.py

Line 97 in eeae19b

x1 = ratio*np.exp(k1)

The test running that line is test_plot_xq2. I've checked and there's no infinite or anything weird there. And anyway the test fails after that. That is, the next test fails.

[RoyStegeman]

And more specifically, the test fails due to this line here

Given that it only fails after, how do you know that causes the issue? Is x1 not calculated correctly?

[scarlehoff]

If I comment that line (and the one below, the problem seems to be np.exp(k1) independently pf the sign) the tests always pass.

If I comment other things the test fails just the same.

But I don't see any issues. When I print it out k1 is just a numpy array with reasonable numbers of order (1).

[Zaharid]

Sounds like we are mutating something unintentionally.

[scarlehoff]

Yep, that's what I thought. And given that function (a few steps removed) acts on dataframes it might be that vp is caching something that it shouldn't.

Probably that line I linked is just a red herring and pushes the dataframe just enough to make it fail so the problem is somewhere else. But I have no idea where.

[siranipour]

Also if you update the pytest dictionaries, that update is shared across future tests that use it. This has caught me a few times before

[scarlehoff]

Even more interesting, if I change that function to:

        x1 = np.exp(k1)
        x = 1.53e-3 + np.random.rand(100)*(0.012-1.53e-3)
        return np.clip(x,a_min=None,a_max=1, out=x), np.concatenate(( q2,q2 ))

The test fails (x1 is computed but never used).

Instead, if I comment out the x1 line, it doesn't fail. If this was C I would be looking for instance k1 and x1 being different sizes and then memory being written out of place. But this is clearly not the case here.
Also, I can do:

        tmp = k1.copy()
        y = np.exp(tmp)

And it fails just the same! Which means the problem is the fact that np.exp is being called?? Indeed

    z = np.exp(4.0)

fails just the same (while np.log or np.sin works). np.exp(4.0, dtype=np.float32) is also ok or even np.exp(4.0, dtype=np.float128). The problem is also for float64. Of course this works:

   z = np.float64(np.exp(4.0, dtype=np.float128)

The version of numpy is the same I have in my computer, so it's not that. The python version is slightly different (3.9.5 in conda vs 3.9.7) but I've updated all usual suspects (matplotlib, pandas, scipy and even hypothesis) and the problem remains.

A solution is to do:

   z = np.expm1(4.0) + 1.0

In any case, the error happens because the covariance_matrix that enters here is full of nan:

nnpdf/validphys2/src/validphys/covmats.py

Line 323 in eeae19b

def sqrt_covmat(covariance_matrix):

Which goes to

nnpdf/validphys2/src/validphys/covmats.py

Line 493 in eeae19b

(I selected an empty line because I don't want to spoil the fun).

Anyway, the punchline is that it seems to be libNNPDF fault. get_covmat() produces infinities if np.exp has been called before. Debugging that was more or less fun, but looking inside libNNPDF crosses the line.

So, if someone takes care of making sure libNNPDF is not used in that module two different issues will be solved, @siranipour could you take care of it?

[siranipour]

Yeah I'll take a look, but I'm a bit confused as to why

    z = np.exp(4.0)

fails the test? How come it doesn't fail for np.log?

[scarlehoff]

It's only calls to a float64 np.exp that make the code fails. Note that even np.expm1 + 1 makes it work.

[siranipour]

Sure, but do we know why that's the case? Seems like that's not our fault